Code Explainers

Code explainers tagged #deduplication

go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
typescript 
type ToastVariant = "info" | "success" | "warning" | "error";
 
interface Toast {
  id: string;

A self-dismissing toast store with dedup

observer-pattern deduplication timers
Intermediate 10 steps
javascript 
class RequestDeduplicator {
  constructor(fetcher) {
    this.fetcher = fetcher;
    this.inFlight = new Map();

Deduplicating in-flight requests in JavaScript

promises caching async
Intermediate 6 steps
ruby 
require "nokogiri"
require "open-uri"
require "set"
 

Extracting and normalizing links in Ruby

web-scraping url-parsing html-parsing
Intermediate 8 steps
java 
@Service
public class WebhookDedupService {
 
    private static final Logger log = LoggerFactory.getLogger(WebhookDedupService.class);

Idempotent webhooks with Redis in Spring

idempotency redis deduplication
Intermediate 7 steps
rust 
use std::collections::HashMap;
use std::future::Future;
use std::hash::Hash;
use std::sync::{Arc, Mutex};

Deduplicating concurrent work in Rust

concurrency deduplication async
Advanced 7 steps
go 
package dedup
 
import (
	"crypto/sha256"

Finding duplicate files by content hash in Go

hashing filesystem deduplication
Intermediate 8 steps
javascript 
import { NextResponse } from 'next/server';
import { Redis } from '@upstash/redis';
 
export const runtime = 'edge';

A deduped page-view counter in Next.js

edge-runtime redis rate-limiting
Intermediate 8 steps
rust 
use itertools::Itertools;
use std::cmp::Ordering;
 
#[derive(Debug, Clone)]

Merging sorted event streams with itertools

iterators merge deduplication
Intermediate 7 steps
go 
package dedup
 
import (
	"hash/fnv"

A rolling Bloom filter deduper in Go

bloom-filter deduplication concurrency
Advanced 7 steps
ruby 
class MailingListDeduplicator
  GMAIL_DOMAINS = %w[gmail.com googlemail.com].freeze
 
  def initialize(subscribers)

Deduplicating a mailing list by canonical email

deduplication normalization service object
Intermediate 8 steps
python 
import hashlib
from collections import defaultdict
from pathlib import Path
 

Finding duplicate files by size then hash

hashing file-io deduplication
Intermediate 7 steps
go 
package handler
 
type flightResult struct {
	status int

Deduping in-flight requests in Gin

middleware concurrency deduplication
Advanced 9 steps
php 
<?php
 
namespace App\Queue;
 

Deduplicating a job queue with Redis

deduplication redis atomicity
Advanced 8 steps
python 
import pandas as pd
import numpy as np
 
 

Cleaning a customer DataFrame with pandas

data-cleaning regex normalization
Intermediate 9 steps
python 
from django.db.models import Q
 
from .models import Order
 

DISTINCT ON queries in Django

orm querysets postgresql
Intermediate 7 steps
rust 
use std::collections::BTreeSet;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Write};
use std::path::Path;

Deduplicating and sorting words in Rust

file-io btreeset string-processing
Intermediate 6 steps
java 
public class RequestCoalescer<K, V> {
 
    private final ConcurrentHashMap<K, CompletableFuture<V>> inFlight = new ConcurrentHashMap<>();
    private final Function<K, V> loader;

Coalescing duplicate requests in Java

concurrency caching completablefuture
Advanced 6 steps