ruby 35 lines · 8 steps

Extracting and normalizing links in Ruby

A small class that fetches a page, pulls every anchor href, and returns clean, absolute, deduplicated URLs.

Explained by highlit
Intermediate web-scraping url-parsing html-parsing deduplication error-handling
1require "nokogiri"
2require "open-uri"
3require "set"
4 
5class LinkExtractor
6 def initialize(url)
7 @base = URI.parse(url)
8 @html = URI.open(url, "User-Agent" => "LinkExtractor/1.0").read
9 @doc = Nokogiri::HTML(@html)
10 end
11 
12 def links
13 @doc.css("a[href]").each_with_object(Set.new) do |anchor, acc|
14 href = anchor["href"].to_s.strip
15 next if href.empty? || href.start_with?("#", "javascript:", "mailto:", "tel:")
16 
17 absolute = normalize(href)
18 acc << absolute if absolute
19 end.to_a
20 end
21 
22 def internal_links
23 links.select { |link| URI.parse(link).host == @base.host }
24 end
25 
26 private
27 
28 def normalize(href)
29 resolved = @base.merge(href)
30 resolved.fragment = nil
31 resolved.to_s
32 rescue URI::InvalidURIError
33 nil
34 end
35end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Resolving relative hrefs against a base URI turns fragile page-local links into usable absolute URLs.
  2. 2Building results into a Set gives you free deduplication before converting back to an array.
  3. 3Wrapping URI parsing in a rescue keeps one malformed link from crashing the whole extraction.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
ruby 
class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable
Intermediate 5 steps
ruby 
class WeeklySignupsReport
  DEFAULT_WEEKS = 12
 
  def initialize(weeks: DEFAULT_WEEKS, source: User.all)

Building a weekly signups report in Rails

service object aggregation group by
Intermediate 7 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Extracting and normalizing links in Ruby — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code