rust 19 lines · 5 steps

Filtering duplicate lines in Rust

A streaming uniq that prints each line only the first time it appears, backed by a HashSet.

Explained by highlit
Beginner stdin hashset buffering deduplication error-propagation
1use std::collections::HashSet;
2use std::io::{self, BufRead, BufWriter, Write};
3 
4fn main() -> io::Result<()> {
5 let stdin = io::stdin();
6 let stdout = io::stdout();
7 let mut out = BufWriter::new(stdout.lock());
8 
9 let mut seen: HashSet<String> = HashSet::new();
10 
11 for line in stdin.lock().lines() {
12 let line = line?;
13 if seen.insert(line.clone()) {
14 writeln!(out, "{}", line)?;
15 }
16 }
17 
18 out.flush()
19}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A HashSet gives O(1) membership tests, and its insert return value tells you whether the value was new.
  2. 2Wrapping stdout in a BufWriter batches writes so per-line output doesn't hammer the OS.
  3. 3Returning io::Result from main lets the ? operator propagate I/O errors without manual match arms.

Related explainers

ruby 
class ContactImporter
  def initialize(rows)
    @rows = rows
  end

Deduplicating contacts with Enumerable#uniq

deduplication normalization enumerable
Intermediate 4 steps
rust 
use std::fmt;
 
#[derive(Clone, Copy, PartialEq, Eq)]
pub struct Money {

Formatting money with Rust's Display trait

trait-implementation integer-arithmetic enums
Intermediate 10 steps
rust 
use axum::{extract::{Query, State}, http::StatusCode, Json};
use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine};
use serde::{Deserialize, Serialize};
use sqlx::PgPool;

Cursor pagination in an Axum handler

cursor-pagination keyset-pagination base64
Advanced 10 steps
rust 
use std::num::ParseIntError;
 
fn parse_line(line: &str) -> Result<Vec<i64>, ParseIntError> {
    line.split(',')

Collecting Results in Rust iterators

iterators error-handling result
Intermediate 7 steps
ruby 
class BulkImporter
  BATCH_SIZE = 500
 
  def initialize(records)

Batching bulk inserts in Rails

batching bulk-insert deduplication
Intermediate 5 steps
rust 
use std::sync::Arc;
 
use axum::{
    extract::State,

Offloading work with a channel in Axum

background-jobs message-passing shared-state
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Filtering duplicate lines in Rust — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code