python 37 lines · 8 steps

Parallel file downloads with a thread pool

A streaming HTTP downloader fanned out across worker threads, collecting results as each finishes.

Explained by highlit
Intermediate concurrency thread-pool streaming-io http error-handling
1import time
2from concurrent.futures import ThreadPoolExecutor, as_completed
3from pathlib import Path
4 
5import requests
6 
7 
8def download_file(url: str, dest_dir: Path, *, chunk_size: int = 8192) -> Path:
9 filename = url.rstrip("/").rsplit("/", 1)[-1] or "index.html"
10 dest = dest_dir / filename
11 with requests.get(url, stream=True, timeout=30) as resp:
12 resp.raise_for_status()
13 with dest.open("wb") as fh:
14 for chunk in resp.iter_content(chunk_size=chunk_size):
15 fh.write(chunk)
16 return dest
17 
18 
19def download_all(urls: list[str], dest_dir: Path, max_workers: int = 8) -> dict[str, Path]:
20 dest_dir.mkdir(parents=True, exist_ok=True)
21 results: dict[str, Path] = {}
22 
23 with ThreadPoolExecutor(max_workers=max_workers) as pool:
24 future_to_url = {
25 pool.submit(download_file, url, dest_dir): url for url in urls
26 }
27 for future in as_completed(future_to_url):
28 url = future_to_url[future]
29 try:
30 path = future.result()
31 except requests.RequestException as exc:
32 print(f"failed {url}: {exc}")
33 continue
34 results[url] = path
35 print(f"saved {url} -> {path}")
36 
37 return results
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Streaming responses with iter_content keeps memory flat regardless of file size.
  2. 2ThreadPoolExecutor is ideal for I/O-bound work like network downloads where threads spend most time waiting.
  3. 3Mapping futures back to their inputs lets you report per-task success or failure without losing context.

Related explainers

go 
package validate
 
import (
	"fmt"

Struct validation with reflection in Go

reflection struct-tags validation
Intermediate 7 steps
ruby 
class ParamCoercer
  TYPES = {
    integer: ->(v) { Integer(v) },
    float:   ->(v) { Float(v) },

Coercing request params by schema in Ruby

lambdas type-coercion lookup-table
Intermediate 7 steps
python 
from datetime import datetime
from zoneinfo import ZoneInfo
 
 

Timezone-safe datetime handling in Python

timezones datetime normalization
Intermediate 4 steps
rust 
use std::env;
use std::time::Duration;
 
#[derive(Debug, Clone)]

Loading typed config from environment variables in Rust

configuration error-handling generics
Intermediate 7 steps
java 
public class EventBus {
 
    private final Map<Class<?>, List<Subscriber>> subscribers = new ConcurrentHashMap<>();
    private final Executor executor;

Building an annotation-driven EventBus in Java

publish-subscribe reflection annotations
Intermediate 7 steps
php 
final class TokenBucketLimiter
{
    private float $tokens;
    private float $lastRefill;

How a token bucket rate limiter works

rate-limiting token-bucket throttling
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Parallel file downloads with a thread pool — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code