ruby 45 lines · 7 steps

Deduplicating uploads by content hash

A service that fingerprints each upload with SHA-256 and reuses an existing record when the bytes already exist.

Explained by highlit
Intermediate hashing deduplication streaming-io dependency-injection
1require "digest"
2 
3class UploadDeduplicator
4 CHUNK_SIZE = 64 * 1024
5 
6 def initialize(store: UploadDigest)
7 @store = store
8 end
9 
10 def register(upload)
11 digest = compute_digest(upload.tempfile.path)
12 
13 existing = @store.find_by(sha256: digest)
14 if existing
15 return DuplicateResult.new(duplicate: true, canonical: existing, digest: digest)
16 end
17 
18 record = @store.create!(
19 sha256: digest,
20 byte_size: upload.size,
21 content_type: upload.content_type,
22 original_filename: upload.original_filename
23 )
24 
25 DuplicateResult.new(duplicate: false, canonical: record, digest: digest)
26 end
27 
28 private
29 
30 def compute_digest(path)
31 sha = Digest::SHA256.new
32 File.open(path, "rb") do |io|
33 while (chunk = io.read(CHUNK_SIZE))
34 sha.update(chunk)
35 end
36 end
37 sha.hexdigest
38 end
39 
40 DuplicateResult = Struct.new(:duplicate, :canonical, :digest, keyword_init: true) do
41 def duplicate?
42 duplicate
43 end
44 end
45end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Content-addressing by cryptographic hash lets you detect identical files regardless of their names or metadata.
  2. 2Reading a file in fixed-size chunks keeps memory flat no matter how large the upload is.
  3. 3Injecting the store dependency and returning a small value object keeps the class easy to test and its result explicit.

Related explainers

typescript 
import { registerLocaleData } from '@angular/common';
import localeFr from '@angular/common/locales/fr';
import localeFrExtra from '@angular/common/locales/extra/fr';
import localeDe from '@angular/common/locales/de';

Locale-aware bootstrapping in Angular

i18n localization dependency-injection
Intermediate 8 steps
ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
typescript 
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import * as Joi from 'joi';
 

Validating env config at boot in NestJS

configuration schema-validation environment-variables
Intermediate 8 steps
rust 
use std::cmp::{Ordering, Reverse};
use std::collections::BinaryHeap;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Lines, Write};

K-way merge of sorted logs in Rust

binary-heap k-way-merge streaming-io
Intermediate 8 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
java 
@Component
@Converter
public class EncryptedStringConverter implements AttributeConverter<String, String> {
 

Transparent column encryption in Spring & JPA

encryption aes-gcm jpa-converter
Advanced 10 steps

Share this explainer

Here's the card — post it anywhere.

Deduplicating uploads by content hash — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code