javascript 39 lines · 7 steps

Paginated APIs with async generators

Stream items from a paginated API lazily using async generators and for-await-of.

Explained by highlit
Advanced async-iterators generators pagination lazy-evaluation promises
1async function* fetchPages(baseUrl, maxPages) {
2 let page = 1;
3 while (page <= maxPages) {
4 const data = await mockFetch(`${baseUrl}?page=${page}`);
5 if (!data.items.length) return;
6 yield* data.items;
7 if (!data.hasMore) return;
8 page++;
9 }
10}
11 
12function mockFetch(url) {
13 return new Promise((resolve) => {
14 const match = url.match(/page=(\d+)/);
15 const page = Number(match[1]);
16 setTimeout(() => {
17 resolve({
18 items: page <= 3 ? [`item-${page}a`, `item-${page}b`] : [],
19 hasMore: page < 3,
20 });
21 }, 10);
22 });
23}
24 
25async function collectAll(baseUrl) {
26 const results = [];
27 for await (const item of fetchPages(baseUrl, 10)) {
28 results.push(item);
29 }
30 return results;
31}
32 
33async function* take(asyncIterable, n) {
34 let count = 0;
35 for await (const value of asyncIterable) {
36 if (count++ >= n) return;
37 yield value;
38 }
39}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Async generators let you model an unbounded paginated source as a flat stream of items, fetching pages only as they're consumed.
  2. 2yield* delegates to an iterable, flattening a page's items into the generator's own output without a manual loop.
  3. 3Because consumption is lazy, a wrapper like take can stop early and the producer simply never fetches the pages it doesn't need.

Related explainers

javascript 
'use server'
 
import { revalidatePath } from 'next/cache'
import { redirect } from 'next/navigation'

How a Next.js Server Action updates a post

server-actions authorization validation
Intermediate 7 steps
typescript 
type RetryOptions = {
  retries?: number;
  timeoutMs?: number;
  baseDelayMs?: number;

Retry with timeout and backoff in TypeScript

promises retry exponential-backoff
Intermediate 10 steps
python 
from collections.abc import Mapping
from typing import Any, Iterator
 
 

Flattening nested config into dotted keys

recursion generators tree-traversal
Intermediate 7 steps
javascript 
const express = require('express');
 
const v1 = express.Router();
 

Versioning an API with Express Routers

api versioning routing modularity
Intermediate 10 steps
python 
import csv
import io
from datetime import datetime
 

Streaming a CSV export in Flask

streaming generators csv
Intermediate 9 steps
javascript 
const RATE_LIMIT = 100;
const WINDOW_MS = 60 * 1000;
const BLOCK_MS = 5 * 60 * 1000;
 

Building a rate-limiting middleware in Express

rate-limiting middleware closures
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Paginated APIs with async generators — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code