rust 40 lines · 8 steps

A parallel map-reduce with scoped threads

Splitting a slice across worker threads, mapping and folding each chunk in parallel, then combining the partial results.

Explained by highlit
Advanced concurrency scoped-threads map-reduce generics trait-bounds
1use std::thread;
2 
3pub fn parallel_map_reduce<T, M, R, F, G, H>(
4 data: &[T],
5 workers: usize,
6 map: F,
7 reduce: G,
8 identity: H,
9) -> R
10where
11 T: Sync,
12 M: Send,
13 R: Send,
14 F: Fn(&T) -> M + Sync,
15 G: Fn(R, M) -> R + Sync,
16 H: Fn() -> R + Sync,
17{
18 if data.is_empty() {
19 return identity();
20 }
21 let workers = workers.max(1).min(data.len());
22 let chunk_size = (data.len() + workers - 1) / workers;
23 
24 let partials: Vec<R> = thread::scope(|scope| {
25 let handles: Vec<_> = data
26 .chunks(chunk_size)
27 .map(|chunk| {
28 scope.spawn(|| {
29 chunk
30 .iter()
31 .map(&map)
32 .fold(identity(), &reduce)
33 })
34 })
35 .collect();
36 handles.into_iter().map(|h| h.join().unwrap()).collect()
37 });
38 
39 partials.into_iter().fold(identity(), &reduce)
40}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Scoped threads let closures borrow stack data directly, avoiding clones or Arc just to share the input slice.
  2. 2An identity-producing closure lets each chunk fold independently, so partials can be combined the same way at the end.
  3. 3Sync and Send bounds encode exactly which values cross thread boundaries, making the parallelism safe at compile time.

Related explainers

rust 
use axum::{extract::{Path, State}, http::StatusCode, Json};
use dashmap::DashMap;
use serde::Serialize;
use std::sync::Arc;

Request coalescing in an Axum handler

caching concurrency request-coalescing
Advanced 8 steps
go 
package config
 
import (
	"fmt"

A thread-safe config singleton in Go

singleton concurrency environment-variables
Intermediate 7 steps
rust 
use std::net::Ipv4Addr;
use std::str::FromStr;
 
#[derive(Debug, Clone, Copy)]

Parsing and matching IPv4 CIDR ranges in Rust

bitwise-operations parsing error-handling
Intermediate 8 steps
typescript 
type CsvColumn<T> = {
  header: string;
  value: (row: T) => string | number | boolean | null | undefined;
};

Building a type-safe CSV writer in TypeScript

generics serialization escaping
Intermediate 7 steps
rust 
use std::sync::OnceLock;
 
static CRC_TABLE: OnceLock<[u32; 256]> = OnceLock::new();
 

Table-driven CRC32 with lazy init in Rust

checksums lazy-initialization lookup-tables
Intermediate 8 steps
rust 
use axum::{
    body::Body,
    extract::Path,
    http::{header, HeaderMap, HeaderValue, StatusCode},

HTTP range requests for video streaming in Axum

http-range-requests streaming async-io
Advanced 8 steps

Share this explainer

Here's the card — post it anywhere.

A parallel map-reduce with scoped threads — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code