php 61 lines · 8 steps

Bounded-concurrency HTTP fetching with curl_multi

A PHP function fetches many URLs in parallel while capping how many requests run at once.

Explained by highlit
Advanced concurrency http curl throttling event-loop
1<?php
2 
3function fetchAll(array $urls, int $concurrency = 10, int $timeout = 15): array
4{
5 $multi = curl_multi_init();
6 $handles = [];
7 $results = [];
8 $pending = $urls;
9 $index = 0;
10 
11 $addHandle = function (int $key, string $url) use ($multi, &$handles, $timeout): void {
12 $ch = curl_init($url);
13 curl_setopt_array($ch, [
14 CURLOPT_RETURNTRANSFER => true,
15 CURLOPT_FOLLOWLOCATION => true,
16 CURLOPT_CONNECTTIMEOUT => 5,
17 CURLOPT_TIMEOUT => $timeout,
18 CURLOPT_USERAGENT => 'multi-fetch/1.0',
19 ]);
20 curl_multi_add_handle($multi, $ch);
21 $handles[(int) $ch] = ['key' => $key, 'handle' => $ch];
22 };
23 
24 while ($index < count($pending) && $index < $concurrency) {
25 $addHandle($index, $pending[$index]);
26 $index++;
27 }
28 
29 do {
30 $status = curl_multi_exec($multi, $running);
31 if ($running) {
32 curl_multi_select($multi, 1.0);
33 }
34 
35 while ($info = curl_multi_info_read($multi)) {
36 $ch = $info['handle'];
37 $meta = $handles[(int) $ch];
38 
39 $results[$meta['key']] = [
40 'url' => $pending[$meta['key']],
41 'status' => (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE),
42 'body' => $info['result'] === CURLE_OK ? curl_multi_getcontent($ch) : null,
43 'error' => $info['result'] === CURLE_OK ? null : curl_error($ch),
44 ];
45 
46 curl_multi_remove_handle($multi, $ch);
47 curl_close($ch);
48 unset($handles[(int) $ch]);
49 
50 if ($index < count($pending)) {
51 $addHandle($index, $pending[$index]);
52 $index++;
53 }
54 }
55 } while ($running || !empty($handles));
56 
57 curl_multi_close($multi);
58 ksort($results);
59 
60 return $results;
61}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A curl multi handle lets you run many transfers on one thread by polling for completion instead of blocking on each request.
  2. 2Capping concurrency and refilling slots as requests finish keeps a fixed number in flight without overwhelming the target.
  3. 3Keying results by the original index lets you launch out of order yet return them in input order via a final sort.

Related explainers

java 
public class SlidingLogRateLimiter {
 
    private final int maxRequests;
    private final long windowMillis;

How a sliding-log rate limiter works

rate-limiting concurrency sliding-window
Advanced 8 steps
go 
package theme
 
import (
	"fmt"

Per-tenant HTML templates with Gin's renderer

multi-tenancy concurrency html-templates
Advanced 8 steps
php 
<?php
 
namespace App\Http\Controllers;
 

Streaming a filtered CSV export in Laravel

streaming csv-export query-builder
Intermediate 9 steps
python 
import time
import threading
from enum import Enum
from functools import wraps

Building a circuit breaker in Python

circuit-breaker resilience decorators
Advanced 7 steps
typescript 
import { Injectable } from '@angular/core';
import { HttpInterceptor, HttpRequest, HttpHandler, HttpEvent, HttpErrorResponse } from '@angular/common/http';
import { BehaviorSubject, Observable, throwError } from 'rxjs';
import { catchError, filter, take, switchMap } from 'rxjs/operators';

Refreshing auth tokens in an Angular interceptor

http-interceptor token-refresh rxjs
Advanced 8 steps
go 
package auth
 
import (
	"net/http"

Setting and reading secure session cookies in Go

cookies session-management security
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

Bounded-concurrency HTTP fetching with curl_multi — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code