javascript 50 lines · 9 steps

Streaming a large JSON export in Express

Stream a database query straight to the HTTP response as chunked JSON without buffering it all in memory.

Explained by highlit
Express Advanced streaming backpressure cursors chunked-transfer json
1const express = require('express');
2const router = express.Router();
3 
4router.get('/exports/orders', async (req, res) => {
5 const { from, to } = req.query;
6 
7 res.status(200);
8 res.setHeader('Content-Type', 'application/json; charset=utf-8');
9 res.setHeader('Transfer-Encoding', 'chunked');
10 res.setHeader('Content-Disposition', 'attachment; filename="orders-export.json"');
11 
12 res.write('[');
13 
14 let first = true;
15 const cursor = Order.find({ createdAt: { $gte: new Date(from), $lte: new Date(to) } })
16 .populate('customer', 'name email')
17 .lean()
18 .cursor({ batchSize: 500 });
19 
20 req.on('close', () => cursor.close());
21 
22 try {
23 for await (const order of cursor) {
24 const chunk = (first ? '' : ',') + JSON.stringify({
25 id: order._id,
26 total: order.total,
27 currency: order.currency,
28 customer: order.customer,
29 placedAt: order.createdAt,
30 });
31 first = false;
32 
33 if (!res.write(chunk)) {
34 await new Promise((resolve) => res.once('drain', resolve));
35 }
36 }
37 
38 res.write(']');
39 res.end();
40 } catch (err) {
41 req.log.error({ err }, 'orders export failed mid-stream');
42 if (!res.headersSent) {
43 res.status(500).json({ error: 'export_failed' });
44 } else {
45 res.destroy(err);
46 }
47 }
48});
49 
50module.exports = router;
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Streaming from a database cursor keeps memory flat regardless of result-set size.
  2. 2Respecting write() backpressure by waiting for 'drain' prevents overwhelming a slow client.
  3. 3Once headers are sent you can no longer send an error status, so mid-stream failures must destroy the connection instead.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
javascript 
import { useState, useEffect, useCallback, useRef } from 'react';
 
const cache = new Map();
const inflight = new Map();

Building a stale-while-revalidate hook in React

caching request-deduplication custom-hooks
Advanced 10 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
javascript 
import { useEffect, useRef, useState } from 'react';
 
export function useDelayedFlag(active, delay = 300) {
  const [visible, setVisible] = useState(false);

Delaying a loading spinner with a React hook

custom-hooks debouncing cleanup
Intermediate 8 steps
javascript 
const SWIPE_THRESHOLD = 80;
const MAX_TRANSLATE = 120;
 
export function attachSwipeToDismiss(element, onDismiss) {

Building a swipe-to-dismiss gesture in JS

touch-events gesture-detection dom-manipulation
Intermediate 10 steps

Share this explainer

Here's the card — post it anywhere.

Streaming a large JSON export in Express — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code