python 31 lines · 7 steps

Finding duplicate files by size then hash

A two-pass scan that groups files by size, then confirms duplicates by hashing only the candidates.

Explained by highlit
Intermediate hashing file-io deduplication streaming defaultdict
1import hashlib
2from collections import defaultdict
3from pathlib import Path
4 
5 
6def _hash_file(path: Path, chunk_size: int = 65536) -> str:
7 digest = hashlib.sha256()
8 with path.open("rb") as fh:
9 for chunk in iter(lambda: fh.read(chunk_size), b""):
10 digest.update(chunk)
11 return digest.hexdigest()
12 
13 
14def find_duplicate_files(root: str | Path) -> dict[str, list[Path]]:
15 root = Path(root)
16 by_size: dict[int, list[Path]] = defaultdict(list)
17 for path in root.rglob("*"):
18 if path.is_file():
19 by_size[path.stat().st_size].append(path)
20 
21 duplicates: dict[str, list[Path]] = defaultdict(list)
22 for size, paths in by_size.items():
23 if len(paths) < 2:
24 continue
25 for path in paths:
26 try:
27 duplicates[_hash_file(path)].append(path)
28 except OSError:
29 continue
30 
31 return {digest: paths for digest, paths in duplicates.items() if len(paths) > 1}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Cheap filters first: grouping by size avoids hashing files that can't possibly match.
  2. 2Reading in fixed chunks keeps memory flat regardless of file size.
  3. 3Guarding I/O with try/except lets a scan survive unreadable files instead of crashing.

Related explainers

javascript 
import { NextResponse } from 'next/server';
import { db } from '@/lib/db';
 
function csvCell(value) {

Streaming a CSV export in a Next.js route

streaming csv readablestream
Advanced 9 steps
python 
from flask import Flask, request, g, jsonify
from flask_babel import Babel, gettext as _, format_datetime
from datetime import datetime
 

Per-request localization in Flask with Babel

i18n content-negotiation request-lifecycle
Intermediate 8 steps
rust 
use std::fs::File;
use std::io::{Read, Seek, SeekFrom};
 
#[derive(Debug)]

Parsing an HTTP Range header in Rust

http-range parsing file-io
Intermediate 10 steps
javascript 
class MovingAverage {
  constructor(windowSize) {
    if (!Number.isInteger(windowSize) || windowSize <= 0) {
      throw new RangeError('windowSize must be a positive integer');

A rolling average over a fixed window

circular-buffer streaming async-generators
Intermediate 7 steps
python 
from django.core.cache import cache
from django.core.cache.utils import make_template_fragment_key
from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver

Busting template fragment caches in Django

caching signals cache-invalidation
Intermediate 4 steps
ruby 
require "charlock_holmes"
 
class TextFileNormalizer
  DEFAULT_CONFIDENCE = 60

Normalizing text files to clean UTF-8 in Ruby

encoding text-processing file-io
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Finding duplicate files by size then hash — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code