python 29 lines · 6 steps

Ranking the top words in a PDF

Extract text from a PDF, tokenize it with a regex, drop stop words, and count what's left.

Explained by highlit

Intermediate text-processing regex counting generators pdf

1import re
2from collections import Counter
3from pathlib import Path
4 
5from pypdf import PdfReader
6 
7WORD_RE = re.compile(r"[a-z]+(?:'[a-z]+)?")
8 
9STOP_WORDS = {
10    "the", "a", "an", "and", "or", "but", "if", "of", "to", "in", "on",
11    "at", "for", "with", "is", "are", "was", "were", "be", "been", "as",
12    "that", "this", "it", "by", "from", "not", "no",
13}
14 
15 
16def extract_text(pdf_path: Path) -> str:
17    reader = PdfReader(pdf_path)
18    pages = (page.extract_text() or "" for page in reader.pages)
19    return "\n".join(pages)
20 
21 
22def top_words(pdf_path: Path, limit: int = 20) -> list[tuple[str, int]]:
23    text = extract_text(pdf_path).lower()
24    counts = Counter(
25        word
26        for word in WORD_RE.findall(text)
27        if word not in STOP_WORDS and len(word) > 2
28    )
29    return counts.most_common(limit)

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Combining a compiled regex with a stop-word set gives you clean, meaningful tokens in one pass.
2Generator expressions let you stream and filter tokens straight into Counter without building intermediate lists.
3Counter.most_common turns a frequency tally into a ranked result with almost no code.

Related explainers

java

public final class IbanValidator {
 
    private static final Map<String, Integer> COUNTRY_LENGTHS = Map.ofEntries(
        Map.entry("DE", 22), Map.entry("FR", 27), Map.entry("GB", 22),

Validating IBANs with the mod-97 checksum

validation checksum modular-arithmetic

Intermediate 8 steps

python

import django_filters
from django import forms
from django.utils import timezone

Building a date-range FilterSet in Django

filtering querysets form-widgets

Intermediate 6 steps

python

import json
import hashlib
from datetime import timedelta

Idempotent payment endpoints in FastAPI

idempotency redis distributed-locking

Advanced 9 steps

python

from flask import Blueprint, jsonify, request, abort
 
v1 = Blueprint("users_v1", __name__)
v2 = Blueprint("users_v2", __name__)

Versioning a Flask API with Blueprints

api versioning blueprints rest

Intermediate 7 steps

python

import time
import threading
from enum import Enum
from functools import wraps

Building a circuit breaker in Python

circuit-breaker resilience decorators

Advanced 7 steps

java

public final class Slugifier {
 
    private static final Pattern NON_LATIN = Pattern.compile("[^\\w-]");
    private static final Pattern WHITESPACE = Pattern.compile("[\\s]+");