rust 52 lines · 8 steps

How a tokenizer turns text into tokens

A peekable character iterator scans arithmetic input and emits a typed token stream, failing cleanly on bad input.

Explained by highlit
Intermediate lexing enums iterators error-handling state-machine
1#[derive(Debug, Clone, PartialEq)]
2pub enum Token {
3 Number(f64),
4 Plus,
5 Minus,
6 Star,
7 Slash,
8 LParen,
9 RParen,
10}
11 
12pub fn tokenize(input: &str) -> Result<Vec<Token>, String> {
13 let mut tokens = Vec::new();
14 let mut chars = input.chars().peekable();
15 
16 while let Some(&c) = chars.peek() {
17 match c {
18 ' ' | '\t' | '\n' | '\r' => {
19 chars.next();
20 }
21 '+' => { chars.next(); tokens.push(Token::Plus); }
22 '-' => { chars.next(); tokens.push(Token::Minus); }
23 '*' => { chars.next(); tokens.push(Token::Star); }
24 '/' => { chars.next(); tokens.push(Token::Slash); }
25 '(' => { chars.next(); tokens.push(Token::LParen); }
26 ')' => { chars.next(); tokens.push(Token::RParen); }
27 '0'..='9' | '.' => {
28 let mut literal = String::new();
29 let mut seen_dot = false;
30 while let Some(&d) = chars.peek() {
31 if d.is_ascii_digit() {
32 literal.push(d);
33 chars.next();
34 } else if d == '.' && !seen_dot {
35 seen_dot = true;
36 literal.push(d);
37 chars.next();
38 } else {
39 break;
40 }
41 }
42 let value = literal
43 .parse::<f64>()
44 .map_err(|_| format!("invalid number: {literal}"))?;
45 tokens.push(Token::Number(value));
46 }
47 other => return Err(format!("unexpected character: {other:?}")),
48 }
49 }
50 
51 Ok(tokens)
52}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Peeking before consuming lets a scanner decide what a character means without losing it.
  2. 2Modeling tokens as a typed enum makes downstream parsing exhaustive and safe.
  3. 3Returning a Result at the tokenizer boundary surfaces malformed input early with context.

Related explainers

python 
from flask import Blueprint, jsonify
from sqlalchemy import text
from sqlalchemy.exc import SQLAlchemyError
 

Building a health check endpoint in Flask

health-check blueprint error-handling
Intermediate 6 steps
go 
package breaker
 
import (
	"errors"

How a circuit breaker works in Go

circuit-breaker state-machine concurrency
Intermediate 8 steps
rust 
use axum::{
    extract::{FromRequestParts, Host},
    http::{request::Parts, StatusCode},
};

Multi-tenant DB routing in Axum extractors

multi-tenancy extractors connection-pooling
Advanced 8 steps
python 
from fastapi import FastAPI, Request, status
from fastapi.encoders import jsonable_encoder
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse

Redacting sensitive fields in FastAPI errors

validation error-handling security
Intermediate 7 steps
rust 
use std::{collections::HashSet, sync::Arc};
 
use axum::{
    body::Body,

Feature-flag middleware in Axum

middleware async shared-state
Advanced 7 steps
rust 
pub fn format_size(bytes: u64) -> String {
    const UNITS: [&str; 7] = ["B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB"];
 
    if bytes < 1024 {

Formatting byte counts as human-readable sizes

bit-manipulation formatting unit-conversion
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

How a tokenizer turns text into tokens — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code