go 58 lines · 8 steps

Finding duplicate files by content hash in Go

Walk a directory tree, hash every file, and group the ones whose contents collide.

Explained by highlit
Intermediate hashing filesystem deduplication sha256 grouping
1package dedup
2 
3import (
4 "crypto/sha256"
5 "encoding/hex"
6 "io"
7 "os"
8 "path/filepath"
9)
10 
11type DuplicateSet struct {
12 Hash string
13 Paths []string
14}
15 
16func FindDuplicates(root string) ([]DuplicateSet, error) {
17 byHash := make(map[string][]string)
18 
19 err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
20 if err != nil {
21 return err
22 }
23 if info.IsDir() || !info.Mode().IsRegular() {
24 return nil
25 }
26 sum, err := hashFile(path)
27 if err != nil {
28 return err
29 }
30 byHash[sum] = append(byHash[sum], path)
31 return nil
32 })
33 if err != nil {
34 return nil, err
35 }
36 
37 var dups []DuplicateSet
38 for sum, paths := range byHash {
39 if len(paths) > 1 {
40 dups = append(dups, DuplicateSet{Hash: sum, Paths: paths})
41 }
42 }
43 return dups, nil
44}
45 
46func hashFile(path string) (string, error) {
47 f, err := os.Open(path)
48 if err != nil {
49 return "", err
50 }
51 defer f.Close()
52 
53 h := sha256.New()
54 if _, err := io.Copy(h, f); err != nil {
55 return "", err
56 }
57 return hex.EncodeToString(h.Sum(nil)), nil
58}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Hashing file contents lets you detect duplicates regardless of filename or location.
  2. 2A map keyed by hash naturally buckets items that share the same value.
  3. 3Streaming a file through io.Copy into a hasher avoids loading it fully into memory.

Related explainers

go 
package streaming
 
import (
	"bufio"

Streaming NDJSON logs over HTTP in Go

http-streaming channels select
Advanced 10 steps
go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
ruby 
class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable
Intermediate 5 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
ruby 
class ApplicationController < ActionController::Base
  EXPERIMENTS = {
    checkout_button_color: %w[control blue green],
    onboarding_flow: %w[control streamlined]

How A/B test cohorts are assigned in Rails

a-b-testing cookies hashing
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Finding duplicate files by content hash in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code