go 40 lines · 6 steps

Streaming a JSON array in Go

Decode a large JSON array element-by-element with a streaming decoder instead of loading it all into memory.

Explained by highlit
Intermediate streaming json decoder error-wrapping io
1package feed
2 
3import (
4 "encoding/json"
5 "fmt"
6 "io"
7)
8 
9type Event struct {
10 ID string `json:"id"`
11 Type string `json:"type"`
12 Payload json.RawMessage `json:"payload"`
13}
14 
15func StreamEvents(r io.Reader, handle func(Event) error) error {
16 dec := json.NewDecoder(r)
17 
18 tok, err := dec.Token()
19 if err != nil {
20 return fmt.Errorf("read opening token: %w", err)
21 }
22 if delim, ok := tok.(json.Delim); !ok || delim != '[' {
23 return fmt.Errorf("expected JSON array, got %v", tok)
24 }
25 
26 for dec.More() {
27 var ev Event
28 if err := dec.Decode(&ev); err != nil {
29 return fmt.Errorf("decode event: %w", err)
30 }
31 if err := handle(ev); err != nil {
32 return fmt.Errorf("handle event %s: %w", ev.ID, err)
33 }
34 }
35 
36 if _, err := dec.Token(); err != nil {
37 return fmt.Errorf("read closing token: %w", err)
38 }
39 return nil
40}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1json.Decoder can walk a document token-by-token, letting you process arbitrarily large arrays without buffering the whole thing.
  2. 2json.RawMessage defers parsing of a field so each consumer can interpret the payload on its own terms.
  3. 3Wrapping errors with %w preserves the underlying cause while adding context at each layer.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
go 
package streaming
 
import (
	"bufio"

Streaming NDJSON logs over HTTP in Go

http-streaming channels select
Advanced 10 steps
go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Streaming a JSON array in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code