#lexing code explainers

typescript

type TokenType = "keyword" | "string" | "comment" | "number" | "text";
 
interface Token {
  type: TokenType;

How a regex tokenizer highlights code

tokenizer regex lexing

Intermediate 10 steps

rust

#[derive(Debug, Clone, PartialEq)]
pub enum Token {
    Number(f64),
    Plus,

How a tokenizer turns text into tokens

lexing enums iterators

Intermediate 8 steps

javascript

const TOKEN_SPECS = [
  ["comment", /^\/\/[^\n]*|^\/\*[\s\S]*?\*\//],
  ["string", /^"(?:\\.|[^"\\])*"|^'(?:\\.|[^'\\])*'|^`(?:\\.|[^`\\])*`/],
  ["number", /^0[xX][\da-fA-F]+|^\d+(?:\.\d+)?(?:[eE][+-]?\d+)?/],

Building a syntax highlighter tokenizer

tokenizer regular-expressions lexing

Intermediate 8 steps