python 29 lines · 6 steps

Ranking the top words in a PDF

Extract text from a PDF, tokenize it with a regex, drop stop words, and count what's left.

Explained by highlit
Intermediate text-processing regex counting generators pdf
1import re
2from collections import Counter
3from pathlib import Path
4 
5from pypdf import PdfReader
6 
7WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
8 
9STOP_WORDS = {
10 "the", "a", "an", "and", "or", "but", "if", "of", "to", "in", "on",
11 "at", "for", "with", "is", "are", "was", "were", "be", "been", "as",
12 "that", "this", "it", "by", "from", "not", "no",
13}
14 
15 
16def extract_text(pdf_path: Path) -> str:
17 reader = PdfReader(pdf_path)
18 pages = (page.extract_text() or "" for page in reader.pages)
19 return "\n".join(pages)
20 
21 
22def top_words(pdf_path: Path, limit: int = 20) -> list[tuple[str, int]]:
23 text = extract_text(pdf_path).lower()
24 counts = Counter(
25 word
26 for word in WORD_RE.findall(text)
27 if word not in STOP_WORDS and len(word) > 2
28 )
29 return counts.most_common(limit)
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Combining a compiled regex with a stop-word set gives you clean, meaningful tokens in one pass.
  2. 2Generator expressions let you stream and filter tokens straight into Counter without building intermediate lists.
  3. 3Counter.most_common turns a frequency tally into a ranked result with almost no code.

Related explainers

python 
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
 
app = FastAPI()
 

Building a WebSocket chat with FastAPI

websockets broadcast connection-management
Intermediate 9 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
python 
import time
import uuid
 
from django.utils.deprecation import MiddlewareMixin

Attaching per-request context in Django

middleware request lifecycle multi-tenancy
Intermediate 7 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
python 
import secrets
 
from django.contrib.auth import authenticate, login
from django.core.cache import cache

Two-factor login with OTP in Django

two-factor-auth one-time-passwords caching
Intermediate 9 steps
ruby 
class SnippetHighlighter
  CONTEXT_RADIUS = 60
  MAX_TERMS = 8
 

Building search-result snippets in Ruby

regular-expressions text-processing search
Intermediate 9 steps

Share this explainer

Here's the card — post it anywhere.

Ranking the top words in a PDF — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code