java 38 lines · 8 steps

Finding duplicate files by content hash

Group files under a directory by their SHA-256 hash to detect duplicates that share identical content.

Explained by highlit
Intermediate hashing file-io streams grouping try-with-resources
1public class DuplicateFinder {
2 
3 public Map<String, List<Path>> findDuplicates(Path root) throws IOException {
4 Map<String, List<Path>> byHash = new HashMap<>();
5 
6 try (Stream<Path> files = Files.walk(root)) {
7 files.filter(Files::isRegularFile)
8 .forEach(path -> {
9 try {
10 String hash = sha256(path);
11 byHash.computeIfAbsent(hash, k -> new ArrayList<>()).add(path);
12 } catch (IOException e) {
13 throw new UncheckedIOException("Failed to hash " + path, e);
14 }
15 });
16 }
17 
18 return byHash.entrySet().stream()
19 .filter(e -> e.getValue().size() > 1)
20 .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
21 }
22 
23 private String sha256(Path path) throws IOException {
24 try {
25 MessageDigest digest = MessageDigest.getInstance("SHA-256");
26 try (InputStream in = Files.newInputStream(path);
27 DigestInputStream dis = new DigestInputStream(in, digest)) {
28 byte[] buffer = new byte[8192];
29 while (dis.read(buffer) != -1) {
30 // drain the stream so the digest consumes every byte
31 }
32 }
33 return HexFormat.of().formatHex(digest.digest());
34 } catch (NoSuchAlgorithmException e) {
35 throw new IllegalStateException("SHA-256 unavailable", e);
36 }
37 }
38}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Content hashing identifies duplicates regardless of filename, since identical bytes produce identical digests.
  2. 2computeIfAbsent cleanly builds a multimap by lazily creating each key's list on first insert.
  3. 3Streaming a file through a DigestInputStream avoids loading large files fully into memory.

Related explainers

java 
@Component
public class OrderRecoveryHandler {
 
    private final RabbitTemplate rabbitTemplate;

Dead-letter recovery with Spring AMQP

dead-letter-queue retry exponential-backoff
Advanced 9 steps
java 
@Component
public class CorrelationIdWebFilter implements WebFilter, Ordered {
 
    public static final String CORRELATION_ID_HEADER = "X-Correlation-Id";

Correlation IDs in a Spring WebFlux filter

reactive web filter correlation id
Advanced 7 steps
java 
public final class DeepCopier {
 
    private static final ObjectMapper MAPPER = new ObjectMapper()
            .registerModule(new JavaTimeModule())

Deep-copying objects via Jackson serialization

serialization deep-copy generics
Intermediate 7 steps
rust 
use std::fs::File;
use std::io::{self, Read};
use std::path::Path;
 

Streaming a SHA-256 hash of a file in Rust

hashing buffered-io streaming
Intermediate 5 steps
javascript 
import { NextResponse } from 'next/server';
import { createWriteStream } from 'node:fs';
import { mkdir } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';

Streaming file uploads in a Next.js route

file-upload streams validation
Intermediate 9 steps
java 
package com.example.time;
 
import java.time.Duration;
import java.time.Instant;

Adding ISO-8601 durations across time zones in Java

date-time iso-8601 parsing
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Finding duplicate files by content hash — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code