go 66 lines · 9 steps

Splitting multi-line logs with a Scanner

A custom bufio.SplitFunc carves a log stream into timestamp-prefixed entries so each can be parsed independently.

Explained by highlit
Intermediate parsing streaming bufio tokenization state-machine
1package logparse
2 
3import (
4 "bufio"
5 "bytes"
6 "fmt"
7 "io"
8 "time"
9)
10 
11type Entry struct {
12 Timestamp time.Time
13 Level string
14 Message string
15}
16 
17var entryPrefix = []byte("[20")
18 
19func splitEntries(data []byte, atEOF bool) (advance int, token []byte, err error) {
20 if atEOF && len(data) == 0 {
21 return 0, nil, nil
22 }
23 start := bytes.Index(data, entryPrefix)
24 if start < 0 {
25 if atEOF {
26 return len(data), nil, nil
27 }
28 return 0, nil, nil
29 }
30 next := bytes.Index(data[start+len(entryPrefix):], entryPrefix)
31 if next < 0 {
32 if atEOF {
33 return len(data), data[start:], nil
34 }
35 return 0, nil, nil
36 }
37 end := start + len(entryPrefix) + next
38 return end, data[start:end], nil
39}
40 
41func Parse(r io.Reader) ([]Entry, error) {
42 sc := bufio.NewScanner(r)
43 sc.Buffer(make([]byte, 0, 64*1024), 1<<20)
44 sc.Split(splitEntries)
45 
46 var entries []Entry
47 for sc.Scan() {
48 raw := bytes.TrimSpace(sc.Bytes())
49 close := bytes.IndexByte(raw, ']')
50 if len(raw) == 0 || raw[0] != '[' || close < 0 {
51 continue
52 }
53 ts, err := time.Parse(time.RFC3339, string(raw[1:close]))
54 if err != nil {
55 return nil, fmt.Errorf("parse timestamp: %w", err)
56 }
57 rest := bytes.TrimSpace(raw[close+1:])
58 level, msg, _ := bytes.Cut(rest, []byte(" "))
59 entries = append(entries, Entry{
60 Timestamp: ts,
61 Level: string(level),
62 Message: string(bytes.TrimSpace(msg)),
63 })
64 }
65 return entries, sc.Err()
66}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A custom bufio.SplitFunc lets you tokenize a stream on arbitrary boundaries instead of just newlines.
  2. 2SplitFunc must signal 'need more data' by returning a zero advance so the Scanner refills its buffer.
  3. 3The atEOF flag is your cue to emit the final trailing token that has no following delimiter.

Related explainers

go 
package streaming
 
import (
	"bufio"

Streaming NDJSON logs over HTTP in Go

http-streaming channels select
Advanced 10 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

Splitting multi-line logs with a Scanner — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code