python 30 lines · 7 steps

Streaming JSON aggregation with ijson

Sum orders from a huge JSON file without loading it into memory, using streaming parsing and exact decimal math.

Explained by highlit
Intermediate streaming json-parsing generators decimal memory-efficiency
1import ijson
2from decimal import Decimal
3 
4 
5def aggregate_orders(path):
6 totals = {}
7 order_count = 0
8 
9 with open(path, "rb") as fh:
10 for order in ijson.items(fh, "item"):
11 status = order.get("status", "unknown")
12 if status == "cancelled":
13 continue
14 
15 amount = Decimal(str(order["amount"]))
16 currency = order["currency"]
17 totals[currency] = totals.get(currency, Decimal("0")) + amount
18 order_count += 1
19 
20 return {
21 "processed": order_count,
22 "totals": {c: str(v) for c, v in totals.items()},
23 }
24 
25 
26def iter_high_value(path, threshold):
27 with open(path, "rb") as fh:
28 for order in ijson.items(fh, "item"):
29 if Decimal(str(order["amount"])) >= threshold:
30 yield order["id"], order["amount"]
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Streaming parsers like ijson let you process arbitrarily large files with constant memory instead of reading them whole.
  2. 2Wrap currency amounts in Decimal via str() to avoid binary float rounding errors in totals.
  3. 3Yielding matches as you scan turns a filter into a lazy pipeline the caller can consume incrementally.

Related explainers

javascript 
import { Suspense } from 'react';
import { getProducts } from '@/lib/api/products';
import { formatPrice } from '@/lib/format';
 

Streaming a product list with Suspense in Next.js

server-components suspense streaming
Intermediate 8 steps
python 
import os
import uuid
 
from flask import Blueprint, current_app, jsonify, request

Handling secure file uploads in Flask

file-upload validation blueprints
Intermediate 8 steps
python 
from django.core.cache import cache
from django.db.models import Count, Sum, Q
from django.utils import timezone
 

Caching a Django dashboard aggregate query

caching aggregation conditional-queries
Intermediate 7 steps
python 
import time
from contextlib import contextmanager
 
 

A reusable timing context manager in Python

context-manager timing generators
Intermediate 5 steps
python 
import os
from datetime import datetime, timezone
 
import jwt

JWT bearer auth as a FastAPI dependency

authentication jwt dependency-injection
Intermediate 7 steps
python 
import logging
import traceback
 
from flask import Blueprint, jsonify, request

Centralized JSON error handling in Flask

error-handling blueprints json-api
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

Streaming JSON aggregation with ijson — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code