go 71 lines · 10 steps

Streaming NDJSON logs over HTTP in Go

An HTTP handler tails a log source and pushes each entry to the client as newline-delimited JSON, with periodic keep-alives.

Explained by highlit
Advanced http-streaming channels select server-sent-data context-cancellation
1package streaming
2 
3import (
4 "bufio"
5 "encoding/json"
6 "fmt"
7 "net/http"
8 "time"
9)
10 
11type LogEntry struct {
12 Timestamp time.Time `json:"timestamp"`
13 Level string `json:"level"`
14 Message string `json:"message"`
15}
16 
17type LogSource interface {
18 Tail(ctx interface{ Done() <-chan struct{} }) <-chan LogEntry
19}
20 
21func (h *LogHandler) StreamLogs(w http.ResponseWriter, r *http.Request) {
22 flusher, ok := w.(http.Flusher)
23 if !ok {
24 http.Error(w, "streaming unsupported", http.StatusInternalServerError)
25 return
26 }
27 
28 w.Header().Set("Content-Type", "application/x-ndjson")
29 w.Header().Set("Transfer-Encoding", "chunked")
30 w.Header().Set("X-Content-Type-Options", "nosniff")
31 w.WriteHeader(http.StatusOK)
32 
33 buf := bufio.NewWriter(w)
34 enc := json.NewEncoder(buf)
35 
36 keepAlive := time.NewTicker(15 * time.Second)
37 defer keepAlive.Stop()
38 
39 entries := h.source.Tail(r.Context())
40 for {
41 select {
42 case <-r.Context().Done():
43 return
44 case <-keepAlive.C:
45 if _, err := buf.WriteString("\n"); err != nil {
46 return
47 }
48 buf.Flush()
49 flusher.Flush()
50 case entry, open := <-entries:
51 if !open {
52 buf.Flush()
53 flusher.Flush()
54 return
55 }
56 if err := enc.Encode(entry); err != nil {
57 return
58 }
59 if err := buf.Flush(); err != nil {
60 return
61 }
62 flusher.Flush()
63 }
64 }
65}
66 
67type LogHandler struct {
68 source LogSource
69}
70 
71var _ = fmt.Sprintf
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Streaming HTTP responses requires flushing buffers so bytes reach the client before the handler returns.
  2. 2A select loop over data, cancellation, and a timer channel is the idiomatic way to multiplex streaming concerns.
  3. 3Watching the request context lets the server stop work promptly when the client disconnects.

Related explainers

go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
go 
package middleware
 
import (
	"net/http"

Per-plan export limits in Gin middleware

middleware rate-limiting authorization
Intermediate 7 steps
go 
package httputil
 
import (
	"net"

Safely extracting the real client IP in Go

security http ip-spoofing
Intermediate 7 steps
rust 
use std::time::Duration;
use tokio::sync::mpsc;
use tokio::time::{interval, MissedTickBehavior};
 

A token-bucket rate limiter in Tokio

rate-limiting channels async
Advanced 8 steps

Share this explainer

Here's the card — post it anywhere.

Streaming NDJSON logs over HTTP in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code