typescript 56 lines · 9 steps

Building a TF-IDF search index

An inverted index maps terms to documents and ranks matches with TF-IDF scoring.

Explained by highlit
Advanced inverted-index tf-idf tokenization information-retrieval ranking
1type Doc = { id: string; title: string; body: string };
2 
3type Posting = { docId: string; frequency: number; positions: number[] };
4 
5const STOP_WORDS = new Set(["a", "an", "the", "and", "or", "of", "to", "in", "is"]);
6 
7function tokenize(text: string): string[] {
8 return text
9 .toLowerCase()
10 .replace(/[^\p{L}\p{N}\s]/gu, " ")
11 .split(/\s+/)
12 .filter((t) => t.length > 1 && !STOP_WORDS.has(t));
13}
14 
15export class SearchIndex {
16 private index = new Map<string, Posting[]>();
17 private docCount = 0;
18 
19 add(doc: Doc): void {
20 this.docCount++;
21 const tokens = tokenize(`${doc.title} ${doc.title} ${doc.body}`);
22 const seen = new Map<string, Posting>();
23 
24 tokens.forEach((token, pos) => {
25 let posting = seen.get(token);
26 if (!posting) {
27 posting = { docId: doc.id, frequency: 0, positions: [] };
28 seen.set(token, posting);
29 const list = this.index.get(token) ?? [];
30 list.push(posting);
31 this.index.set(token, list);
32 }
33 posting.frequency++;
34 posting.positions.push(pos);
35 });
36 }
37 
38 search(query: string): { docId: string; score: number }[] {
39 const terms = tokenize(query);
40 const scores = new Map<string, number>();
41 
42 for (const term of terms) {
43 const postings = this.index.get(term);
44 if (!postings) continue;
45 const idf = Math.log(1 + this.docCount / postings.length);
46 for (const { docId, frequency } of postings) {
47 const tf = 1 + Math.log(frequency);
48 scores.set(docId, (scores.get(docId) ?? 0) + tf * idf);
49 }
50 }
51 
52 return [...scores.entries()]
53 .map(([docId, score]) => ({ docId, score }))
54 .sort((a, b) => b.score - a.score);
55 }
56}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1An inverted index flips documents into a term-to-postings map so lookups touch only relevant docs.
  2. 2TF-IDF balances how often a term appears against how rare it is across the corpus.
  3. 3Weighting title tokens by repeating them is a cheap way to boost their relevance.

Related explainers

typescript 
import { registerLocaleData } from '@angular/common';
import localeFr from '@angular/common/locales/fr';
import localeFrExtra from '@angular/common/locales/extra/fr';
import localeDe from '@angular/common/locales/de';

Locale-aware bootstrapping in Angular

i18n localization dependency-injection
Intermediate 8 steps
typescript 
import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import * as Joi from 'joi';
 

Validating env config at boot in NestJS

configuration schema-validation environment-variables
Intermediate 8 steps
javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
typescript 
import { Inject, Injectable, Logger } from '@nestjs/common';
import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Cache } from 'cache-manager';
import { InjectRepository } from '@nestjs/typeorm';

A cache-aside country lookup in NestJS

cache-aside dependency-injection batch-lookup
Intermediate 8 steps
typescript 
import { Injectable, effect, signal, computed } from '@angular/core';
 
interface Preferences {
  theme: 'light' | 'dark';

A signal-based preferences store in Angular

signals state-management persistence
Intermediate 7 steps
typescript 
import { useEffect, useState } from "react";
 
interface Section {
  id: string;

Building a scroll-spy hook in React

custom-hooks intersectionobserver dom-observation
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Building a TF-IDF search index — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code