python 37 lines · 8 steps

Parallel file downloads with a thread pool

A streaming HTTP downloader fanned out across worker threads, collecting results as each finishes.

Explained by highlit

Intermediate concurrency thread-pool streaming-io http error-handling

1import time
2from concurrent.futures import ThreadPoolExecutor, as_completed
3from pathlib import Path
4 
5import requests
6 
7 
8def download_file(url: str, dest_dir: Path, *, chunk_size: int = 8192) -> Path:
9    filename = url.rstrip("/").rsplit("/", 1)[-1] or "index.html"
10    dest = dest_dir / filename
11    with requests.get(url, stream=True, timeout=30) as resp:
12        resp.raise_for_status()
13        with dest.open("wb") as fh:
14            for chunk in resp.iter_content(chunk_size=chunk_size):
15                fh.write(chunk)
16    return dest
17 
18 
19def download_all(urls: list[str], dest_dir: Path, max_workers: int = 8) -> dict[str, Path]:
20    dest_dir.mkdir(parents=True, exist_ok=True)
21    results: dict[str, Path] = {}
22 
23    with ThreadPoolExecutor(max_workers=max_workers) as pool:
24        future_to_url = {
25            pool.submit(download_file, url, dest_dir): url for url in urls
26        }
27        for future in as_completed(future_to_url):
28            url = future_to_url[future]
29            try:
30                path = future.result()
31            except requests.RequestException as exc:
32                print(f"failed {url}: {exc}")
33                continue
34            results[url] = path
35            print(f"saved {url} -> {path}")
36 
37    return results

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Streaming responses with iter_content keeps memory flat regardless of file size.
2ThreadPoolExecutor is ideal for I/O-bound work like network downloads where threads spend most time waiting.
3Mapping futures back to their inputs lets you report per-task success or failure without losing context.

Related explainers

rust

use axum::{
    body::{Body, Bytes},
    extract::Request,
    http::StatusCode,

Logging request and response sizes in Axum

middleware http streaming-bodies

Advanced 8 steps

python

from functools import wraps
 
from flask import Blueprint, abort, jsonify
from flask_login import current_user, login_required

Building an admin-only decorator in Flask

decorators authorization access-control

Intermediate 7 steps

package handler
 
type flightResult struct {
	status int

Deduping in-flight requests in Gin

middleware concurrency deduplication

Advanced 9 steps

python

from django.urls import path, include
 
app_name = "api"

How URL-namespaced API versioning works in Django

api versioning url routing namespaces

Intermediate 8 steps

type CreateUserInput struct {
	Name     string `json:"name" binding:"required,min=2,max=64"`
	Email    string `json:"email" binding:"required,email"`
	Password string `json:"password" binding:"required,min=8"`

Turning Gin validation errors into JSON

validation request-binding error-handling

Intermediate 9 steps

javascript

const express = require('express');
const { createProxyMiddleware, fixRequestBody } = require('http-proxy-middleware');
 
const router = express.Router();