go 40 lines · 6 steps

Streaming a JSON array in Go

Decode a large JSON array element-by-element with a streaming decoder instead of loading it all into memory.

Explained by highlit
Intermediate streaming json decoder error-wrapping io
1package feed
2 
3import (
4 "encoding/json"
5 "fmt"
6 "io"
7)
8 
9type Event struct {
10 ID string `json:"id"`
11 Type string `json:"type"`
12 Payload json.RawMessage `json:"payload"`
13}
14 
15func StreamEvents(r io.Reader, handle func(Event) error) error {
16 dec := json.NewDecoder(r)
17 
18 tok, err := dec.Token()
19 if err != nil {
20 return fmt.Errorf("read opening token: %w", err)
21 }
22 if delim, ok := tok.(json.Delim); !ok || delim != '[' {
23 return fmt.Errorf("expected JSON array, got %v", tok)
24 }
25 
26 for dec.More() {
27 var ev Event
28 if err := dec.Decode(&ev); err != nil {
29 return fmt.Errorf("decode event: %w", err)
30 }
31 if err := handle(ev); err != nil {
32 return fmt.Errorf("handle event %s: %w", ev.ID, err)
33 }
34 }
35 
36 if _, err := dec.Token(); err != nil {
37 return fmt.Errorf("read closing token: %w", err)
38 }
39 return nil
40}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1json.Decoder can walk a document token-by-token, letting you process arbitrarily large arrays without buffering the whole thing.
  2. 2json.RawMessage defers parsing of a field so each consumer can interpret the payload on its own terms.
  3. 3Wrapping errors with %w preserves the underlying cause while adding context at each layer.

Related explainers

go 
package pipeline
 
import "sync"
 

The fan-in pattern in Go channels

concurrency channels fan-in
Advanced 8 steps
rust 
use axum::{extract::Multipart, http::StatusCode, Json};
use serde::Serialize;
use tokio::io::AsyncWriteExt;
use uuid::Uuid;

Streaming multipart uploads in Axum

multipart streaming async-io
Intermediate 10 steps
go 
package validate
 
import (
	"fmt"

Struct validation with reflection in Go

reflection struct-tags validation
Intermediate 7 steps
go 
package middleware
 
import (
	"bytes"

Verifying webhook HMAC signatures in Gin

hmac middleware webhooks
Intermediate 7 steps
python 
import shutil
import uuid
from pathlib import Path
 

Safe avatar uploads in FastAPI

file-upload validation streaming
Intermediate 8 steps
java 
public class OrderSerializer {
 
    private final ObjectMapper mapper;
 

Configuring a reusable Jackson ObjectMapper

serialization json builder-pattern
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Streaming a JSON array in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code