rust 29 lines · 6 steps

Deduplicating and sorting words in Rust

A BTreeSet collects unique, cleaned words from a file and writes them back out in sorted order.

Explained by highlit
Intermediate file-io btreeset string-processing error-handling deduplication
1use std::collections::BTreeSet;
2use std::fs::File;
3use std::io::{self, BufRead, BufReader, BufWriter, Write};
4use std::path::Path;
5 
6pub fn dedup_sort_words(input: &Path, output: &Path) -> io::Result<usize> {
7 let reader = BufReader::new(File::open(input)?);
8 let mut words: BTreeSet<String> = BTreeSet::new();
9 
10 for line in reader.lines() {
11 let line = line?;
12 for token in line.split_whitespace() {
13 let cleaned: String = token
14 .trim_matches(|c: char| !c.is_alphanumeric())
15 .to_lowercase();
16 if !cleaned.is_empty() {
17 words.insert(cleaned);
18 }
19 }
20 }
21 
22 let mut writer = BufWriter::new(File::create(output)?);
23 for word in &words {
24 writeln!(writer, "{word}")?;
25 }
26 writer.flush()?;
27 
28 Ok(words.len())
29}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A BTreeSet gives you deduplication and sorted order in one data structure, for free.
  2. 2The `?` operator lets each fallible IO call bubble errors up cleanly to the caller.
  3. 3Buffered readers and writers avoid a syscall per line, which matters for large files.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
rust 
use std::cmp::{Ordering, Reverse};
use std::collections::BinaryHeap;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Lines, Write};

K-way merge of sorted logs in Rust

binary-heap k-way-merge streaming-io
Intermediate 8 steps
rust 
use axum::{
    extract::{Path, State},
    response::sse::{Event, KeepAlive, Sse},
};

Streaming import progress with SSE in Axum

server-sent-events streams watch-channel
Advanced 7 steps
rust 
use std::f64::consts::PI;
 
#[derive(Debug, Clone, Copy)]
pub struct LatLng {

Building geographic bounding boxes in Rust

geospatial value-types option
Intermediate 7 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Deduplicating and sorting words in Rust — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code