rust 40 lines · 8 steps

A parallel map-reduce with scoped threads

Splitting a slice across worker threads, mapping and folding each chunk in parallel, then combining the partial results.

Explained by highlit
Advanced concurrency scoped-threads map-reduce generics trait-bounds
1use std::thread;
2 
3pub fn parallel_map_reduce<T, M, R, F, G, H>(
4 data: &[T],
5 workers: usize,
6 map: F,
7 reduce: G,
8 identity: H,
9) -> R
10where
11 T: Sync,
12 M: Send,
13 R: Send,
14 F: Fn(&T) -> M + Sync,
15 G: Fn(R, M) -> R + Sync,
16 H: Fn() -> R + Sync,
17{
18 if data.is_empty() {
19 return identity();
20 }
21 let workers = workers.max(1).min(data.len());
22 let chunk_size = (data.len() + workers - 1) / workers;
23 
24 let partials: Vec<R> = thread::scope(|scope| {
25 let handles: Vec<_> = data
26 .chunks(chunk_size)
27 .map(|chunk| {
28 scope.spawn(|| {
29 chunk
30 .iter()
31 .map(&map)
32 .fold(identity(), &reduce)
33 })
34 })
35 .collect();
36 handles.into_iter().map(|h| h.join().unwrap()).collect()
37 });
38 
39 partials.into_iter().fold(identity(), &reduce)
40}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Scoped threads let closures borrow stack data directly, avoiding clones or Arc just to share the input slice.
  2. 2An identity-producing closure lets each chunk fold independently, so partials can be combined the same way at the end.
  3. 3Sync and Send bounds encode exactly which values cross thread boundaries, making the parallelism safe at compile time.

Related explainers

rust 
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::thread;
 

Aggregating metrics across threads in Rust

concurrency shared-state mutex
Intermediate 7 steps
java 
public class ThumbnailProcessor {
 
    private static final int MAX_CONCURRENCY = 4;
 

Bounded parallel thumbnail rendering in Java

concurrency thread-pool futures
Intermediate 7 steps
rust 
use std::sync::{mpsc, Arc, Mutex};
use std::thread;
use std::time::Duration;
 

Building a thread pool in Rust

concurrency channels thread-pool
Advanced 9 steps
go 
package cache
 
import (
	"container/list"

Building a generic LRU cache in Go

lru-cache generics linked-list
Intermediate 8 steps
rust 
use std::time::Instant;
use tracing::info;
 
pub struct Timer {

A scope-guard timer with Drop in Rust

raii drop-guard timing
Intermediate 7 steps
rust 
use std::collections::HashMap;
 
pub struct Memoizer<K, V, F> {
    cache: HashMap<K, V>,

A generic memoizer in Rust

memoization generics caching
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

A parallel map-reduce with scoped threads — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code