rust 34 lines · 7 steps

Merging sorted event streams with itertools

Two functions combine pre-sorted Rust iterators by timestamp, one deduplicating and one resolving ties by priority.

Explained by highlit
Intermediate iterators merge deduplication generics ordering
1use itertools::Itertools;
2use std::cmp::Ordering;
3 
4#[derive(Debug, Clone)]
5struct Event {
6 timestamp: u64,
7 source: &'static str,
8 payload: String,
9}
10 
11pub fn merge_event_streams<A, B>(primary: A, replica: B) -> impl Iterator<Item = Event>
12where
13 A: IntoIterator<Item = Event>,
14 B: IntoIterator<Item = Event>,
15{
16 primary
17 .into_iter()
18 .merge_by(replica, |left, right| left.timestamp <= right.timestamp)
19 .dedup_by(|a, b| a.timestamp == b.timestamp && a.source == b.source)
20}
21 
22pub fn merge_by_priority<A, B>(high: A, low: B) -> impl Iterator<Item = Event>
23where
24 A: IntoIterator<Item = Event>,
25 B: IntoIterator<Item = Event>,
26{
27 high.into_iter().merge_join_by(low, |l, r| {
28 match l.timestamp.cmp(&r.timestamp) {
29 Ordering::Equal => Ordering::Less,
30 other => other,
31 }
32 })
33 .map(|either| either.into_left().unwrap_or_else(|r| r))
34}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1itertools' merge_by and merge_join_by interleave already-sorted streams lazily without collecting into a buffer.
  2. 2Returning impl Iterator lets a function hand back a composed pipeline while hiding its concrete type.
  3. 3The comparator you pass decides ordering and tie-breaking, giving you fine control over which source wins.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
rust 
use std::cmp::{Ordering, Reverse};
use std::collections::BinaryHeap;
use std::fs::File;
use std::io::{self, BufRead, BufReader, BufWriter, Lines, Write};

K-way merge of sorted logs in Rust

binary-heap k-way-merge streaming-io
Intermediate 8 steps
rust 
use axum::{
    extract::{Path, State},
    response::sse::{Event, KeepAlive, Sse},
};

Streaming import progress with SSE in Axum

server-sent-events streams watch-channel
Advanced 7 steps
rust 
use std::f64::consts::PI;
 
#[derive(Debug, Clone, Copy)]
pub struct LatLng {

Building geographic bounding boxes in Rust

geospatial value-types option
Intermediate 7 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Merging sorted event streams with itertools — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code