python 40 lines · 10 steps

Parsing and applying a robots.txt file

A small class parses robots.txt directives into per-agent groups and decides whether a URL may be crawled by longest-match.

Explained by highlit
Intermediate parsing longest-prefix-match state-machine web-crawling
1from urllib.parse import urlparse
2 
3 
4class RobotsRules:
5 def __init__(self, user_agent="*"):
6 self.user_agent = user_agent
7 self._groups = {}
8 
9 @classmethod
10 def parse(cls, text, user_agent="*"):
11 rules = cls(user_agent)
12 current_agents = []
13 for raw in text.splitlines():
14 line = raw.split("#", 1)[0].strip()
15 if not line or ":" not in line:
16 continue
17 field, value = (part.strip() for part in line.split(":", 1))
18 field = field.lower()
19 if field == "user-agent":
20 current_agents = [value.lower()]
21 elif field in ("allow", "disallow") and current_agents:
22 for agent in current_agents:
23 rules._groups.setdefault(agent, []).append((field, value))
24 return rules
25 
26 def _directives(self):
27 agent = self.user_agent.lower()
28 if agent in self._groups:
29 return self._groups[agent]
30 return self._groups.get("*", [])
31 
32 def is_allowed(self, url):
33 path = urlparse(url).path or "/"
34 best_match = (-1, True)
35 for field, pattern in self._directives():
36 if not pattern:
37 continue
38 if path.startswith(pattern) and len(pattern) > best_match[0]:
39 best_match = (len(pattern), field == "allow")
40 return best_match[1]
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Grouping directives by user-agent keeps the parse output ready for fast per-agent lookup.
  2. 2The robots.txt spec resolves conflicts by longest matching path, so track the winner as you scan.
  3. 3Stripping comments and skipping malformed lines makes a parser robust against messy real-world input.

Related explainers

python 
from configparser import ConfigParser, ExtendedInterpolation
from pathlib import Path
 
 

Layered INI config loading in Python

configuration parsing defaults
Intermediate 8 steps
python 
from pathlib import Path
 
from django.contrib.auth.decorators import login_required
from django.http import FileResponse, Http404, HttpResponseForbidden

Serving files with access checks in Django

authorization http-caching file-serving
Intermediate 7 steps
rust 
use once_cell::sync::Lazy;
use regex::Regex;
 
static LOG_LINE: Lazy<Regex> = Lazy::new(|| {

Parsing access logs with a lazy regex in Rust

regex lazy-initialization parsing
Intermediate 7 steps
python 
from copy import deepcopy
 
 
class CowSnapshot:

Copy-on-write attribute snapshots in Python

copy-on-write descriptors attribute-interception
Advanced 7 steps
typescript 
type Semver = {
  major: number;
  minor: number;
  patch: number;

Parsing and comparing semver strings in TypeScript

parsing regular-expressions comparison
Intermediate 9 steps
ruby 
class CircuitBreaker
  class OpenCircuitError < StandardError; end
 
  def initialize(failure_threshold: 5, reset_timeout: 30, half_open_max: 1)

How a circuit breaker guards failing calls

state-machine fault-tolerance concurrency
Advanced 9 steps

Share this explainer

Here's the card — post it anywhere.

Parsing and applying a robots.txt file — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code