#tokenizer code explainers

python

class Parser:
    def __init__(self, text):
        self.tokens = self._tokenize(text)
        self.pos = 0

A recursive descent arithmetic parser

recursive-descent tokenizer operator-precedence

Intermediate 9 steps

typescript

type TokenType = "keyword" | "string" | "comment" | "number" | "text";
 
interface Token {
  type: TokenType;

How a regex tokenizer highlights code

tokenizer regex lexing

Intermediate 10 steps

javascript

const TOKEN_SPECS = [
  ["comment", /^\/\/[^\n]*|^\/\*[\s\S]*?\*\//],
  ["string", /^"(?:\\.|[^"\\])*"|^'(?:\\.|[^'\\])*'|^`(?:\\.|[^`\\])*`/],
  ["number", /^0[xX][\da-fA-F]+|^\d+(?:\.\d+)?(?:[eE][+-]?\d+)?/],

Building a syntax highlighter tokenizer

tokenizer regular-expressions lexing

Intermediate 8 steps