php 61 lines · 8 steps

Bounded-concurrency HTTP fetching with curl_multi

A PHP function fetches many URLs in parallel while capping how many requests run at once.

Explained by highlit
Advanced concurrency http curl throttling event-loop
1<?php
2 
3function fetchAll(array $urls, int $concurrency = 10, int $timeout = 15): array
4{
5 $multi = curl_multi_init();
6 $handles = [];
7 $results = [];
8 $pending = $urls;
9 $index = 0;
10 
11 $addHandle = function (int $key, string $url) use ($multi, &$handles, $timeout): void {
12 $ch = curl_init($url);
13 curl_setopt_array($ch, [
14 CURLOPT_RETURNTRANSFER => true,
15 CURLOPT_FOLLOWLOCATION => true,
16 CURLOPT_CONNECTTIMEOUT => 5,
17 CURLOPT_TIMEOUT => $timeout,
18 CURLOPT_USERAGENT => 'multi-fetch/1.0',
19 ]);
20 curl_multi_add_handle($multi, $ch);
21 $handles[(int) $ch] = ['key' => $key, 'handle' => $ch];
22 };
23 
24 while ($index < count($pending) && $index < $concurrency) {
25 $addHandle($index, $pending[$index]);
26 $index++;
27 }
28 
29 do {
30 $status = curl_multi_exec($multi, $running);
31 if ($running) {
32 curl_multi_select($multi, 1.0);
33 }
34 
35 while ($info = curl_multi_info_read($multi)) {
36 $ch = $info['handle'];
37 $meta = $handles[(int) $ch];
38 
39 $results[$meta['key']] = [
40 'url' => $pending[$meta['key']],
41 'status' => (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE),
42 'body' => $info['result'] === CURLE_OK ? curl_multi_getcontent($ch) : null,
43 'error' => $info['result'] === CURLE_OK ? null : curl_error($ch),
44 ];
45 
46 curl_multi_remove_handle($multi, $ch);
47 curl_close($ch);
48 unset($handles[(int) $ch]);
49 
50 if ($index < count($pending)) {
51 $addHandle($index, $pending[$index]);
52 $index++;
53 }
54 }
55 } while ($running || !empty($handles));
56 
57 curl_multi_close($multi);
58 ksort($results);
59 
60 return $results;
61}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A curl multi handle lets you run many transfers on one thread by polling for completion instead of blocking on each request.
  2. 2Capping concurrency and refilling slots as requests finish keeps a fixed number in flight without overwhelming the target.
  3. 3Keying results by the original index lets you launch out of order yet return them in input order via a final sort.

Related explainers

php 
<?php
 
class NameParser
{

Parsing a full name into components in PHP

string-parsing arrays normalization
Intermediate 8 steps
php 
<?php
 
namespace App\Services\Checkout;
 

Validating coupons with Laravel's Pipeline

pipeline chain of responsibility transactions
Intermediate 7 steps
php 
<?php
 
namespace App\Services;
 

How a password strength validator works in PHP

validation regular-expressions data-driven
Intermediate 8 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
php 
<?php
 
namespace App\Services;
 

Building a cached daily leaderboard in Laravel

caching aggregation eager-loading
Intermediate 9 steps
java 
public static Map<String, String> parseCookieHeader(String header) {
    Map<String, String> cookies = new LinkedHashMap<>();
    if (header == null || header.isBlank()) {
        return cookies;

Parsing an HTTP Cookie header in Java

string-parsing http url-decoding
Intermediate 6 steps

Share this explainer

Here's the card — post it anywhere.

Bounded-concurrency HTTP fetching with curl_multi — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code