python 54 lines · 9 steps

Finding near-duplicate images by perceptual hash

Cluster visually similar images by comparing perceptual hashes within a distance threshold, then prune the redundant copies.

Explained by highlit
Intermediate perceptual-hashing clustering hamming-distance deduplication filesystem
1from pathlib import Path
2from collections import defaultdict
3 
4from PIL import Image
5imagehash = __import__("imagehash")
6 
7 
8def _phash(path: Path) -> imagehash.ImageHash:
9 with Image.open(path) as img:
10 return imagehash.phash(img.convert("RGB"), hash_size=16)
11 
12 
13def find_near_duplicates(directory, max_distance=8):
14 exts = {".jpg", ".jpeg", ".png", ".webp", ".bmp", ".gif"}
15 hashes = {}
16 for path in sorted(Path(directory).rglob("*")):
17 if path.suffix.lower() not in exts:
18 continue
19 try:
20 hashes[path] = _phash(path)
21 except (OSError, ValueError):
22 continue
23 
24 buckets = defaultdict(list)
25 for path, h in hashes.items():
26 prefix = str(h)[:8]
27 buckets[prefix].append(path)
28 
29 seen = set()
30 groups = []
31 for path, h in hashes.items():
32 if path in seen:
33 continue
34 cluster = [path]
35 seen.add(path)
36 for other, oh in hashes.items():
37 if other in seen:
38 continue
39 if h - oh <= max_distance:
40 cluster.append(other)
41 seen.add(other)
42 if len(cluster) > 1:
43 groups.append(sorted(cluster, key=lambda p: (-p.stat().st_size, p.name)))
44 return groups
45 
46 
47def prune_duplicates(directory, max_distance=8, dry_run=True):
48 removed = []
49 for keep, *dupes in find_near_duplicates(directory, max_distance):
50 for dup in dupes:
51 removed.append(dup)
52 if not dry_run:
53 dup.unlink()
54 return removed
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Perceptual hashes let you measure image similarity numerically instead of demanding byte-for-byte equality.
  2. 2Hamming distance between hashes plus a threshold turns fuzzy 'looks alike' into a concrete grouping rule.
  3. 3Separating detection from deletion — with a dry-run default — makes destructive operations safe to preview.

Related explainers

python 
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
from starlette.status import HTTP_413_REQUEST_ENTITY_TOO_LARGE

Enforcing a max request body size in FastAPI

middleware streaming request-limits
Advanced 6 steps
python 
import logging
import uuid
from contextvars import ContextVar
 

Request ID tracing in FastAPI middleware

middleware context-variables request-tracing
Intermediate 7 steps
python 
import json
import time
import queue
 

Server-Sent Events streaming in Flask

server-sent-events streaming pub-sub
Advanced 9 steps
python 
from flask import Blueprint, request, jsonify
from marshmallow import Schema, fields, validate, ValidationError, EXCLUDE
 
from .models import db, User

How a Flask blueprint validates and creates users

validation rest-api schema
Intermediate 8 steps
python 
import random
 
import click
from faker import Faker

Building a Flask seed command with Click

cli database seeding orm
Intermediate 7 steps
python 
import smtplib
from email.message import EmailMessage
from threading import Thread
 

Sending welcome emails off the request thread in Flask

background-threads app-context email
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Finding near-duplicate images by perceptual hash — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code