go 32 lines · 5 steps

Generic deduplication in Go

Two generic functions strip duplicates from a slice while preserving order, using a set tracked by an empty-struct map.

Explained by highlit
Intermediate generics deduplication maps slices order-preserving
1package collection
2 
3func Deduplicate[T comparable](items []T) []T {
4 seen := make(map[T]struct{}, len(items))
5 result := make([]T, 0, len(items))
6 
7 for _, item := range items {
8 if _, ok := seen[item]; ok {
9 continue
10 }
11 seen[item] = struct{}{}
12 result = append(result, item)
13 }
14 
15 return result
16}
17 
18func DeduplicateBy[T any, K comparable](items []T, key func(T) K) []T {
19 seen := make(map[K]struct{}, len(items))
20 result := make([]T, 0, len(items))
21 
22 for _, item := range items {
23 k := key(item)
24 if _, ok := seen[k]; ok {
25 continue
26 }
27 seen[k] = struct{}{}
28 result = append(result, item)
29 }
30 
31 return result
32}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A map with struct{} values is Go's idiomatic zero-memory set for membership checks.
  2. 2Pre-sizing maps and slices with the input length avoids repeated reallocation during the pass.
  3. 3Adding a key function turns an exact-equality dedup into one keyed on any derived comparable value.

Related explainers

go 
package streaming
 
import (
	"bufio"

Streaming NDJSON logs over HTTP in Go

http-streaming channels select
Advanced 10 steps
go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
go 
package middleware
 
import (
	"net/http"

Per-plan export limits in Gin middleware

middleware rate-limiting authorization
Intermediate 7 steps
rust 
use axum::{
    async_trait,
    extract::{rejection::JsonRejection, FromRequest, Request},
    http::StatusCode,

A validated JSON extractor in Axum

extractors validation error-handling
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Generic deduplication in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code