python 36 lines · 8 steps

Bounded-concurrency HTTP fetching with asyncio

Fetch many URLs in parallel while capping in-flight requests with a semaphore, returning a structured result for each.

Explained by highlit
Intermediate async concurrency semaphore http error-handling
1import asyncio
2from dataclasses import dataclass
3 
4import aiohttp
5 
6 
7@dataclass
8class FetchResult:
9 url: str
10 status: int | None
11 body: str | None
12 error: str | None = None
13 
14 
15async def fetch_one(
16 session: aiohttp.ClientSession,
17 semaphore: asyncio.Semaphore,
18 url: str,
19 timeout: float = 10.0,
20) -> FetchResult:
21 async with semaphore:
22 try:
23 async with session.get(url, timeout=aiohttp.ClientTimeout(total=timeout)) as resp:
24 return FetchResult(url=url, status=resp.status, body=await resp.text())
25 except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
26 return FetchResult(url=url, status=None, body=None, error=str(exc))
27 
28 
29async def fetch_all(urls: list[str], max_concurrency: int = 8) -> list[FetchResult]:
30 semaphore = asyncio.Semaphore(max_concurrency)
31 async with aiohttp.ClientSession() as session:
32 tasks = [
33 asyncio.create_task(fetch_one(session, semaphore, url))
34 for url in urls
35 ]
36 return await asyncio.gather(*tasks)
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A shared asyncio.Semaphore caps how many coroutines run their critical section at once, throttling concurrency without blocking the loop.
  2. 2Returning a result object per URL instead of raising lets partial failures coexist with successes in a single gathered list.
  3. 3Reusing one ClientSession across all requests amortizes connection pooling and TLS setup for every fetch.

Related explainers

javascript 
const express = require('express');
const { createProxyMiddleware, fixRequestBody } = require('http-proxy-middleware');
 
const router = express.Router();

Building an API gateway proxy in Express

reverse-proxy middleware api-gateway
Intermediate 6 steps
rust 
use axum::{
    extract::{FromRequestParts, Query},
    http::{request::Parts, StatusCode},
};

Building a custom Axum extractor for query filters

extractors query-parsing enums
Intermediate 8 steps
go 
package middleware
 
import (
	"net/http"

Per-IP write rate limiting in Gin

rate-limiting middleware concurrency
Intermediate 8 steps
rust 
use std::sync::Arc;
use axum::{
    extract::{Multipart, State},
    http::StatusCode,

Throttling file uploads in Axum with a Semaphore

concurrency backpressure multipart
Advanced 8 steps
go 
package middleware
 
import (
	"net/http"

A content-type guard middleware in Gin

middleware closures http
Intermediate 7 steps
python 
from pathlib import Path
from typing import Iterator
 
 

Filtering files with pathlib.glob

generators filesystem filtering
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

Bounded-concurrency HTTP fetching with asyncio — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code