ruby 45 lines · 8 steps

A thread-pool URL fetcher in Ruby

A fixed pool of worker threads drains a shared queue of URLs, fetching them concurrently while capping how many run at once.

Explained by highlit
Intermediate concurrency thread-pool queues http error-handling
1require "net/http"
2require "json"
3 
4class UrlFetcher
5 Result = Struct.new(:url, :status, :body, :error, keyword_init: true)
6 
7 def initialize(concurrency: 8, timeout: 10)
8 @concurrency = concurrency
9 @timeout = timeout
10 end
11 
12 def fetch_all(urls)
13 queue = Queue.new
14 urls.each { |url| queue << url }
15 results = Queue.new
16 
17 workers = Array.new([@concurrency, urls.size].min) do
18 Thread.new do
19 until queue.empty?
20 url = queue.pop(true) rescue break
21 results << fetch_one(url)
22 end
23 end
24 end
25 
26 workers.each(&:join)
27 Array.new(results.size) { results.pop }
28 end
29 
30 private
31 
32 def fetch_one(url)
33 uri = URI.parse(url)
34 response = Net::HTTP.start(uri.host, uri.port,
35 use_ssl: uri.scheme == "https",
36 open_timeout: @timeout,
37 read_timeout: @timeout) do |http|
38 http.request(Net::HTTP::Get.new(uri))
39 end
40 
41 Result.new(url: url, status: response.code.to_i, body: response.body)
42 rescue => e
43 Result.new(url: url, status: nil, error: e.message)
44 end
45end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A thread-safe Queue lets you distribute work across a fixed pool without manual locking.
  2. 2Capping worker count to min(concurrency, size) avoids spawning idle threads for tiny inputs.
  3. 3Rescuing per-request keeps one failed fetch from taking down the whole batch.

Related explainers

ruby 
class Debouncer
  def initialize(delay:)
    @delay = delay
    @mutex = Mutex.new

A thread-safe debouncer in Ruby

concurrency debouncing mutex
Advanced 5 steps
rust 
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::thread;
 

A round-robin worker pool in Rust

concurrency thread-pool channels
Intermediate 7 steps
go 
package auth
 
import (
	"errors"

Hashing and verifying passwords with bcrypt in Go

password-hashing bcrypt error-handling
Intermediate 7 steps
typescript 
import { useState, useCallback } from 'react';
 
type Todo = {
  id: string;

Optimistic UI updates in a React hook

optimistic-updates custom-hooks error-handling
Intermediate 8 steps
ruby 
class SyncContactsJob < ApplicationJob
  queue_as :external_api
 
  MAX_CONCURRENCY = 5

Rate-limited CRM sync in a Rails job

background-jobs rate-limiting redis
Advanced 9 steps
rust 
use std::fs::File;
use std::io::{self, Read};
use std::path::Path;
 

Streaming a SHA-256 hash of a file in Rust

hashing buffered-io streaming
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

A thread-pool URL fetcher in Ruby — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code