java 38 lines · 8 steps

Finding duplicate files by content hash

Group files under a directory by their SHA-256 hash to detect duplicates that share identical content.

Explained by highlit

Intermediate hashing file-io streams grouping try-with-resources

1public class DuplicateFinder {
2 
  public Map<String, List<Path>> findDuplicates(Path root) throws IOException {
      Map<String, List<Path>> byHash = new HashMap<>();
5 
      try (Stream<Path> files = Files.walk(root)) {
          files.filter(Files::isRegularFile)
               .forEach(path -> {
                   try {
                       String hash = sha256(path);
                       byHash.computeIfAbsent(hash, k -> new ArrayList<>()).add(path);
                   } catch (IOException e) {
                       throw new UncheckedIOException("Failed to hash " + path, e);
                   }
               });
      }
17 
      return byHash.entrySet().stream()
              .filter(e -> e.getValue().size() > 1)
              .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
  }
22 
  private String sha256(Path path) throws IOException {
      try {
          MessageDigest digest = MessageDigest.getInstance("SHA-256");
          try (InputStream in = Files.newInputStream(path);
               DigestInputStream dis = new DigestInputStream(in, digest)) {
              byte[] buffer = new byte[8192];
              while (dis.read(buffer) != -1) {
                  // drain the stream so the digest consumes every byte
              }
          }
          return HexFormat.of().formatHex(digest.digest());
      } catch (NoSuchAlgorithmException e) {
          throw new IllegalStateException("SHA-256 unavailable", e);
      }
  }
38}

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Content hashing identifies duplicates regardless of filename, since identical bytes produce identical digests.
2computeIfAbsent cleanly builds a multimap by lazily creating each key's list on first insert.
3Streaming a file through a DigestInputStream avoids loading large files fully into memory.

Related explainers

java

@Component
@Converter
public class EncryptedStringConverter implements AttributeConverter<String, String> {

Transparent column encryption in Spring & JPA

encryption aes-gcm jpa-converter

Advanced 10 steps

rust

use axum::{
    extract::{Path, State},
    response::sse::{Event, KeepAlive, Sse},
};

Streaming import progress with SSE in Axum

server-sent-events streams watch-channel

Advanced 7 steps

ruby

class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable

Intermediate 5 steps

java

package com.acme.billing.config;
 
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;

Feature-flagged beans with Spring @ConditionalOnProperty

feature-flags conditional-beans strategy-pattern

Intermediate 5 steps

java

public static Map<String, String> parseCookieHeader(String header) {
    Map<String, String> cookies = new LinkedHashMap<>();
    if (header == null || header.isBlank()) {
        return cookies;

Parsing an HTTP Cookie header in Java

string-parsing http url-decoding

Intermediate 6 steps

ruby

class ApplicationController < ActionController::Base
  EXPERIMENTS = {
    checkout_button_color: %w[control blue green],
    onboarding_flow: %w[control streamlined]