ruby 50 lines · 7 steps

Paginating an HTTP API with a Ruby enumerator

A client walks every page of a cursor-paginated REST API and yields each record lazily.

Explained by highlit

Intermediate pagination http enumerator rest-api authentication

1require "net/http"
2require "json"
3require "uri"
4require "base64"
5 
6class GitHubStarsClient
7  BASE = "https://api.example.com/v1"
8 
9  def initialize(token:, page_size: 100)
10    @token = token
11    @page_size = page_size
12  end
13 
14  def each_repository
15    return enum_for(:each_repository) unless block_given?
16 
17    cursor = nil
18    loop do
19      page = fetch_page(cursor)
20      page.fetch("data").each { |repo| yield repo }
21 
22      cursor = page.dig("page_info", "next_cursor")
23      break unless page.dig("page_info", "has_next") && cursor
24    end
25  end
26 
27  private
28 
29  def fetch_page(cursor)
30    query = { "limit" => @page_size }
31    query["cursor"] = cursor if cursor
32 
33    uri = URI("#{BASE}/repositories")
34    uri.query = URI.encode_www_form(query)
35 
36    request = Net::HTTP::Get.new(uri)
37    request["Authorization"] = "Bearer #{@token}"
38    request["Accept"] = "application/json"
39 
40    response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: uri.scheme == "https") do |http|
41      http.request(request)
42    end
43 
44    unless response.is_a?(Net::HTTPSuccess)
45      raise "API request failed: #{response.code} #{response.message}"
46    end
47 
48    JSON.parse(response.body)
49  end
50end

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Returning enum_for when no block is given lets a method act as both an iterator and a lazy enumerator.
2Cursor-based pagination loops until the API signals there is no next page, carrying the cursor between requests.
3Checking the response type before parsing turns silent HTTP failures into explicit, actionable errors.

Related explainers

ruby

require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection

Intermediate 7 steps

ruby

class UserAgentParser
  BROWSERS = [
    [/Edg\/([\d.]+)/, "Edge"],
    [/OPR\/([\d.]+)/, "Opera"],

Parsing user-agent strings in Ruby

regex pattern-matching lookup-tables

Intermediate 8 steps

python

import time
import uuid
 
from django.utils.deprecation import MiddlewareMixin

Attaching per-request context in Django

middleware request lifecycle multi-tenancy

Intermediate 7 steps

ruby

class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable

Intermediate 5 steps

ruby

class WeeklySignupsReport
  DEFAULT_WEEKS = 12
 
  def initialize(weeks: DEFAULT_WEEKS, source: User.all)

Building a weekly signups report in Rails

service object aggregation group by

Intermediate 7 steps

java

public static Map<String, String> parseCookieHeader(String header) {
    Map<String, String> cookies = new LinkedHashMap<>();
    if (header == null || header.isBlank()) {
        return cookies;