ruby 45 lines · 8 steps

A thread-pool URL fetcher in Ruby

A fixed pool of worker threads drains a shared queue of URLs, fetching them concurrently while capping how many run at once.

Explained by highlit
Intermediate concurrency thread-pool queues http error-handling
1require "net/http"
2require "json"
3 
4class UrlFetcher
5 Result = Struct.new(:url, :status, :body, :error, keyword_init: true)
6 
7 def initialize(concurrency: 8, timeout: 10)
8 @concurrency = concurrency
9 @timeout = timeout
10 end
11 
12 def fetch_all(urls)
13 queue = Queue.new
14 urls.each { |url| queue << url }
15 results = Queue.new
16 
17 workers = Array.new([@concurrency, urls.size].min) do
18 Thread.new do
19 until queue.empty?
20 url = queue.pop(true) rescue break
21 results << fetch_one(url)
22 end
23 end
24 end
25 
26 workers.each(&:join)
27 Array.new(results.size) { results.pop }
28 end
29 
30 private
31 
32 def fetch_one(url)
33 uri = URI.parse(url)
34 response = Net::HTTP.start(uri.host, uri.port,
35 use_ssl: uri.scheme == "https",
36 open_timeout: @timeout,
37 read_timeout: @timeout) do |http|
38 http.request(Net::HTTP::Get.new(uri))
39 end
40 
41 Result.new(url: url, status: response.code.to_i, body: response.body)
42 rescue => e
43 Result.new(url: url, status: nil, error: e.message)
44 end
45end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A thread-safe Queue lets you distribute work across a fixed pool without manual locking.
  2. 2Capping worker count to min(concurrency, size) avoids spawning idle threads for tiny inputs.
  3. 3Rescuing per-request keeps one failed fetch from taking down the whole batch.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
ruby 
class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables
Intermediate 8 steps
ruby 
class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable
Intermediate 5 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
ruby 
class WeeklySignupsReport
  DEFAULT_WEEKS = 12
 
  def initialize(weeks: DEFAULT_WEEKS, source: User.all)

Building a weekly signups report in Rails

service object aggregation group by
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

A thread-pool URL fetcher in Ruby — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code