java 41 lines · 6 steps

Three ways to deduplicate a list by key in Java

A small utility class showing distinct-by-key, latest-per-key, and a custom collector that all dedupe streams without overriding equals.

Explained by highlit

Intermediate streams deduplication collectors generics functional-interfaces

1import java.util.ArrayList;
2import java.util.Comparator;
3import java.util.HashSet;
4import java.util.List;
5import java.util.Set;
6import java.util.function.Function;
7import java.util.stream.Collector;
8import java.util.stream.Collectors;
9 
10public final class Deduplication {
11 
12    public static <T, K> List<T> distinctByKey(List<T> items, Function<? super T, ? extends K> keyExtractor) {
13        Set<K> seen = new HashSet<>();
14        return items.stream()
15                .filter(item -> seen.add(keyExtractor.apply(item)))
16                .collect(Collectors.toList());
17    }
18 
19    public static <T, K> List<T> latestPerKey(
20            List<T> items,
21            Function<? super T, ? extends K> keyExtractor,
22            Comparator<? super T> recency) {
23        return new ArrayList<>(items.stream()
24                .collect(Collectors.toMap(
25                        keyExtractor,
26                        Function.identity(),
27                        (existing, candidate) -> recency.compare(candidate, existing) >= 0 ? candidate : existing))
28                .values());
29    }
30 
31    public static <T> Collector<T, ?, List<T>> distinctByKeyCollector(Function<? super T, ?> keyExtractor) {
32        return Collector.of(
33                () -> new java.util.LinkedHashMap<Object, T>(),
34                (map, item) -> map.putIfAbsent(keyExtractor.apply(item), item),
35                (left, right) -> {
36                    right.forEach(left::putIfAbsent);
37                    return left;
38                },
39                map -> new ArrayList<>(map.values()));
40    }
41}

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1A HashSet's add return value doubles as a cheap first-seen test inside a stream filter.
2Collectors.toMap with a merge function lets you resolve key collisions by any rule you choose, like recency.
3A custom Collector can preserve insertion order while deduplicating, something toSet and toMap can't both guarantee.

Related explainers

java

public interface GitHubClient {
 
    @GetExchange("/users/{username}")
    GitHubUser getUser(@PathVariable String username);

Declarative HTTP clients in Spring

http-client declarative-api proxy

Intermediate 8 steps

java

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

Building a trie for autocomplete in Java

trie prefix-tree recursion

Intermediate 8 steps

typescript

type CsvColumn<T> = {
  header: string;
  value: (row: T) => string | number | boolean | null | undefined;
};

Building a type-safe CSV writer in TypeScript

generics serialization escaping

Intermediate 7 steps

java

@RestController
@RequestMapping("/api/products")
public class ProductSearchController {

Binding collection query params in Spring

rest-api query-parameters dependency-injection

Intermediate 6 steps

java

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.Map;

Evaluating math expressions with two stacks

stacks parsing operator-precedence

Intermediate 9 steps

java

@Entity
@Table(name = "orders")
@SQLDelete(sql = "UPDATE orders SET deleted = true, deleted_at = now() WHERE id = ?")
@Where(clause = "deleted = false")