go 66 lines · 9 steps

Splitting multi-line logs with a Scanner

A custom bufio.SplitFunc carves a log stream into timestamp-prefixed entries so each can be parsed independently.

Explained by highlit
Intermediate parsing streaming bufio tokenization state-machine
1package logparse
2 
3import (
4 "bufio"
5 "bytes"
6 "fmt"
7 "io"
8 "time"
9)
10 
11type Entry struct {
12 Timestamp time.Time
13 Level string
14 Message string
15}
16 
17var entryPrefix = []byte("[20")
18 
19func splitEntries(data []byte, atEOF bool) (advance int, token []byte, err error) {
20 if atEOF && len(data) == 0 {
21 return 0, nil, nil
22 }
23 start := bytes.Index(data, entryPrefix)
24 if start < 0 {
25 if atEOF {
26 return len(data), nil, nil
27 }
28 return 0, nil, nil
29 }
30 next := bytes.Index(data[start+len(entryPrefix):], entryPrefix)
31 if next < 0 {
32 if atEOF {
33 return len(data), data[start:], nil
34 }
35 return 0, nil, nil
36 }
37 end := start + len(entryPrefix) + next
38 return end, data[start:end], nil
39}
40 
41func Parse(r io.Reader) ([]Entry, error) {
42 sc := bufio.NewScanner(r)
43 sc.Buffer(make([]byte, 0, 64*1024), 1<<20)
44 sc.Split(splitEntries)
45 
46 var entries []Entry
47 for sc.Scan() {
48 raw := bytes.TrimSpace(sc.Bytes())
49 close := bytes.IndexByte(raw, ']')
50 if len(raw) == 0 || raw[0] != '[' || close < 0 {
51 continue
52 }
53 ts, err := time.Parse(time.RFC3339, string(raw[1:close]))
54 if err != nil {
55 return nil, fmt.Errorf("parse timestamp: %w", err)
56 }
57 rest := bytes.TrimSpace(raw[close+1:])
58 level, msg, _ := bytes.Cut(rest, []byte(" "))
59 entries = append(entries, Entry{
60 Timestamp: ts,
61 Level: string(level),
62 Message: string(bytes.TrimSpace(msg)),
63 })
64 }
65 return entries, sc.Err()
66}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A custom bufio.SplitFunc lets you tokenize a stream on arbitrary boundaries instead of just newlines.
  2. 2SplitFunc must signal 'need more data' by returning a zero advance so the Scanner refills its buffer.
  3. 3The atEOF flag is your cue to emit the final trailing token that has no following delimiter.

Related explainers

rust 
use std::time::Duration;
 
#[derive(Debug, PartialEq)]
pub enum ParseDurationError {

Parsing duration strings safely in Rust

parsing error-handling checked-arithmetic
Intermediate 8 steps
typescript 
type TokenType = "keyword" | "string" | "comment" | "number" | "text";
 
interface Token {
  type: TokenType;

How a regex tokenizer highlights code

tokenizer regex lexing
Intermediate 10 steps
go 
package handlers
 
import (
	"net/http"

Custom validators and binding in Gin

validation struct-tags error-handling
Intermediate 8 steps
javascript 
function parseHexColor(hex) {
  const cleaned = hex.trim().replace(/^#/, '');
 
  const expand = (short) =>

Parsing hex colors into RGBA channels

parsing bitwise regex
Intermediate 7 steps
go 
package hashring
 
import (
	"hash/crc32"

How consistent hashing works in Go

consistent-hashing load-balancing concurrency
Intermediate 8 steps
python 
import hashlib
from collections import defaultdict
from pathlib import Path
 

Finding duplicate files by size then hash

hashing file-io deduplication
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Splitting multi-line logs with a Scanner — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code