ruby 45 lines · 7 steps

Deduplicating uploads by content hash

A service that fingerprints each upload with SHA-256 and reuses an existing record when the bytes already exist.

Explained by highlit

Intermediate hashing deduplication streaming-io dependency-injection

1require "digest"
2 
3class UploadDeduplicator
CHUNK_SIZE = 64 * 1024
5 
def initialize(store: UploadDigest)
  @store = store
end
9 
def register(upload)
  digest = compute_digest(upload.tempfile.path)
12 
  existing = @store.find_by(sha256: digest)
  if existing
    return DuplicateResult.new(duplicate: true, canonical: existing, digest: digest)
  end
17 
  record = @store.create!(
    sha256: digest,
    byte_size: upload.size,
    content_type: upload.content_type,
    original_filename: upload.original_filename
  )
24 
  DuplicateResult.new(duplicate: false, canonical: record, digest: digest)
end
27 
private
29 
def compute_digest(path)
  sha = Digest::SHA256.new
  File.open(path, "rb") do |io|
    while (chunk = io.read(CHUNK_SIZE))
      sha.update(chunk)
    end
  end
  sha.hexdigest
end
39 
DuplicateResult = Struct.new(:duplicate, :canonical, :digest, keyword_init: true) do
  def duplicate?
    duplicate
  end
end
45end

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Content-addressing by cryptographic hash lets you detect identical files regardless of their names or metadata.
2Reading a file in fixed-size chunks keeps memory flat no matter how large the upload is.
3Injecting the store dependency and returning a small value object keeps the class easy to test and its result explicit.

Related explainers

ruby

class Order
  class InvalidTransition < StandardError; end
 
  TRANSITIONS = {

A state machine for order transitions in Ruby

state-machine data-driven error-handling

Intermediate 8 steps

typescript

import { Injectable, UnauthorizedException } from '@nestjs/common';
import { PassportStrategy } from '@nestjs/passport';
import { ExtractJwt, Strategy } from 'passport-jwt';
import { ConfigService } from '@nestjs/config';

How a JWT strategy authenticates in NestJS

authentication jwt passport

Intermediate 7 steps

ruby

class CreateOrderItems < ActiveRecord::Migration[7.1]
  def change
    create_table :order_items do |t|
      t.references :order, null: false, foreign_key: { on_delete: :cascade }

Enforcing order-item integrity in Rails

migrations foreign-keys validations

Intermediate 7 steps

ruby

class Registration < ApplicationRecord
  belongs_to :event
 
  validates :email, presence: true, format: { with: URI::MailTo::EMAIL_REGEXP }

Validating registrations in Rails

validations i18n error-handling

Intermediate 8 steps

python

import os
from datetime import timedelta

Class-based config in a Flask app factory

configuration app-factory inheritance

Intermediate 7 steps

ruby

class ApacheLogParser
  LINE_PATTERN = /\A(?<ip>\S+)\s\S+\s\S+\s\[(?<time>[^\]]+)\]\s"(?<method>[A-Z]+)\s(?<path>\S+)\s(?<protocol>[^"]+)"\s(?<status>\d{3})\s(?<bytes>\d+|-)/
 
  TIME_FORMAT = "%d/%b/%Y:%H:%M:%S %z"