python 33 lines · 7 steps

Two ways to dedupe while keeping order

Stream items through a set to keep first occurrences, or through an OrderedDict to keep the last.

Explained by highlit
Intermediate deduplication generators ordered-data type-hints higher-order-functions
1from collections import OrderedDict
2from typing import Callable, Hashable, Iterable, Iterator, TypeVar
3 
4T = TypeVar("T")
5 
6 
7def dedupe(
8 items: Iterable[T],
9 key: Callable[[T], Hashable] | None = None,
10) -> Iterator[T]:
11 seen: set[Hashable] = set()
12 extract = key or (lambda item: item)
13 
14 for item in items:
15 marker = extract(item)
16 if marker in seen:
17 continue
18 seen.add(marker)
19 yield item
20 
21 
22def dedupe_last_wins(
23 items: Iterable[T],
24 key: Callable[[T], Hashable],
25) -> list[T]:
26 collapsed: OrderedDict[Hashable, T] = OrderedDict()
27 
28 for item in items:
29 marker = key(item)
30 collapsed.pop(marker, None)
31 collapsed[marker] = item
32 
33 return list(collapsed.values())
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A set of computed keys lets you dedupe by identity while still yielding the original objects.
  2. 2Generators make first-wins deduplication lazy, so you never materialize the full input.
  3. 3Popping then re-inserting into an OrderedDict moves a key to the end, giving clean last-wins ordering.

Related explainers

python 
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
 
app = FastAPI()
 

Building a WebSocket chat with FastAPI

websockets broadcast connection-management
Intermediate 9 steps
python 
import time
import uuid
 
from django.utils.deprecation import MiddlewareMixin

Attaching per-request context in Django

middleware request lifecycle multi-tenancy
Intermediate 7 steps
php 
<?php
 
namespace App\Services;
 

How a password strength validator works in PHP

validation regular-expressions data-driven
Intermediate 8 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
python 
import secrets
 
from django.contrib.auth import authenticate, login
from django.core.cache import cache

Two-factor login with OTP in Django

two-factor-auth one-time-passwords caching
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Two ways to dedupe while keeping order — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code