python 30 lines · 7 steps

Streaming JSON aggregation with ijson

Sum orders from a huge JSON file without loading it into memory, using streaming parsing and exact decimal math.

Explained by highlit
Intermediate streaming json-parsing generators decimal memory-efficiency
1import ijson
2from decimal import Decimal
3 
4 
5def aggregate_orders(path):
6 totals = {}
7 order_count = 0
8 
9 with open(path, "rb") as fh:
10 for order in ijson.items(fh, "item"):
11 status = order.get("status", "unknown")
12 if status == "cancelled":
13 continue
14 
15 amount = Decimal(str(order["amount"]))
16 currency = order["currency"]
17 totals[currency] = totals.get(currency, Decimal("0")) + amount
18 order_count += 1
19 
20 return {
21 "processed": order_count,
22 "totals": {c: str(v) for c, v in totals.items()},
23 }
24 
25 
26def iter_high_value(path, threshold):
27 with open(path, "rb") as fh:
28 for order in ijson.items(fh, "item"):
29 if Decimal(str(order["amount"])) >= threshold:
30 yield order["id"], order["amount"]
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Streaming parsers like ijson let you process arbitrarily large files with constant memory instead of reading them whole.
  2. 2Wrap currency amounts in Decimal via str() to avoid binary float rounding errors in totals.
  3. 3Yielding matches as you scan turns a filter into a lazy pipeline the caller can consume incrementally.

Related explainers

python 
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
 
app = FastAPI()
 

Building a WebSocket chat with FastAPI

websockets broadcast connection-management
Intermediate 9 steps
python 
import time
import uuid
 
from django.utils.deprecation import MiddlewareMixin

Attaching per-request context in Django

middleware request lifecycle multi-tenancy
Intermediate 7 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
python 
import secrets
 
from django.contrib.auth import authenticate, login
from django.core.cache import cache

Two-factor login with OTP in Django

two-factor-auth one-time-passwords caching
Intermediate 9 steps
python 
import re
from functools import total_ordering
from typing import Optional
 

Parsing and comparing semantic versions

regex operator-overloading sorting
Intermediate 7 steps
python 
from typing import Any, Sequence, Mapping
 
 
def render_markdown_table(

Rendering an aligned Markdown table in Python

string formatting data transformation closures
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Streaming JSON aggregation with ijson — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code