python 33 lines · 7 steps

Two ways to dedupe while keeping order

Stream items through a set to keep first occurrences, or through an OrderedDict to keep the last.

Explained by highlit
Intermediate deduplication generators ordered-data type-hints higher-order-functions
1from collections import OrderedDict
2from typing import Callable, Hashable, Iterable, Iterator, TypeVar
3 
4T = TypeVar("T")
5 
6 
7def dedupe(
8 items: Iterable[T],
9 key: Callable[[T], Hashable] | None = None,
10) -> Iterator[T]:
11 seen: set[Hashable] = set()
12 extract = key or (lambda item: item)
13 
14 for item in items:
15 marker = extract(item)
16 if marker in seen:
17 continue
18 seen.add(marker)
19 yield item
20 
21 
22def dedupe_last_wins(
23 items: Iterable[T],
24 key: Callable[[T], Hashable],
25) -> list[T]:
26 collapsed: OrderedDict[Hashable, T] = OrderedDict()
27 
28 for item in items:
29 marker = key(item)
30 collapsed.pop(marker, None)
31 collapsed[marker] = item
32 
33 return list(collapsed.values())
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A set of computed keys lets you dedupe by identity while still yielding the original objects.
  2. 2Generators make first-wins deduplication lazy, so you never materialize the full input.
  3. 3Popping then re-inserting into an OrderedDict moves a key to the end, giving clean last-wins ordering.

Related explainers

python 
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, status
from pydantic import BaseModel, EmailStr
from sqlalchemy.orm import Session
 

Building a signup endpoint in FastAPI

dependency-injection request-validation background-tasks
Intermediate 8 steps
python 
from django import forms
from django.utils import timezone
 
from .models import Reservation

Multi-field validation in a Django ModelForm

form validation cross-field validation modelform
Intermediate 7 steps
python 
from django import template
from django.urls import reverse, NoReverseMatch
from django.utils.html import format_html
 

Active nav-link template tags in Django

template tags url routing active state
Intermediate 7 steps
python 
from functools import wraps
import asyncio
 
from fastapi import APIRouter, FastAPI, Request

Per-route request timeouts in FastAPI

decorators async timeouts
Intermediate 6 steps
python 
import os
from pathlib import Path
 
BASE_DIR = Path(__file__).resolve().parent.parent.parent

How a Django settings module is wired

configuration environment-variables middleware
Intermediate 8 steps
python 
def is_valid_card_number(number: str) -> bool:
    digits = [int(c) for c in number if c.isdigit()]
 
    if len(digits) < 13 or len(digits) > 19:

Validating card numbers with the Luhn check

checksum validation luhn-algorithm
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

Two ways to dedupe while keeping order — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code