rust 19 lines · 5 steps

Filtering duplicate lines in Rust

A streaming uniq that prints each line only the first time it appears, backed by a HashSet.

Explained by highlit
Beginner stdin hashset buffering deduplication error-propagation
1use std::collections::HashSet;
2use std::io::{self, BufRead, BufWriter, Write};
3 
4fn main() -> io::Result<()> {
5 let stdin = io::stdin();
6 let stdout = io::stdout();
7 let mut out = BufWriter::new(stdout.lock());
8 
9 let mut seen: HashSet<String> = HashSet::new();
10 
11 for line in stdin.lock().lines() {
12 let line = line?;
13 if seen.insert(line.clone()) {
14 writeln!(out, "{}", line)?;
15 }
16 }
17 
18 out.flush()
19}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A HashSet gives O(1) membership tests, and its insert return value tells you whether the value was new.
  2. 2Wrapping stdout in a BufWriter batches writes so per-line output doesn't hammer the OS.
  3. 3Returning io::Result from main lets the ? operator propagate I/O errors without manual match arms.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
rust 
use std::cmp::{Ordering, Reverse};
use std::collections::BinaryHeap;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Lines, Write};

K-way merge of sorted logs in Rust

binary-heap k-way-merge streaming-io
Intermediate 8 steps
rust 
use axum::{
    extract::{Path, State},
    response::sse::{Event, KeepAlive, Sse},
};

Streaming import progress with SSE in Axum

server-sent-events streams watch-channel
Advanced 7 steps
rust 
use std::f64::consts::PI;
 
#[derive(Debug, Clone, Copy)]
pub struct LatLng {

Building geographic bounding boxes in Rust

geospatial value-types option
Intermediate 7 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Filtering duplicate lines in Rust — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code