Code Explainers

Code explainers tagged #tokenization

javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
typescript 
type Doc = { id: string; title: string; body: string };
 
type Posting = { docId: string; frequency: number; positions: number[] };
 

Building a TF-IDF search index

inverted-index tf-idf tokenization
Advanced 9 steps
go 
package logparse
 
import (
	"bufio"

Splitting multi-line logs with a Scanner

parsing streaming bufio
Intermediate 9 steps
typescript 
interface ParsedName {
  first: string;
  middle: string;
  last: string;

Parsing human names into structured parts

parsing string-manipulation normalization
Intermediate 9 steps
ruby 
class QueryParser
  def self.parse(query_string)
    new(query_string).parse
  end

Parsing nested query strings in Ruby

parsing recursion tokenization
Intermediate 9 steps