Code Explainers

Code explainers tagged #tokenizer

javascript 
const TOKEN_SPECS = [
  ["comment", /^\/\/[^\n]*|^\/\*[\s\S]*?\*\//],
  ["string", /^"(?:\\.|[^"\\])*"|^'(?:\\.|[^'\\])*'|^`(?:\\.|[^`\\])*`/],
  ["number", /^0[xX][\da-fA-F]+|^\d+(?:\.\d+)?(?:[eE][+-]?\d+)?/],

Building a syntax highlighter tokenizer

tokenizer regular-expressions lexing
Intermediate 8 steps