go 61 lines · 9 steps

Bounded-concurrency HTTP fetching in Go

Fetch many URLs in parallel while capping in-flight requests with a semaphore channel, honoring context cancellation.

Explained by highlit
Advanced concurrency semaphore context-cancellation goroutines waitgroup
1package fetcher
2 
3import (
4 "context"
5 "fmt"
6 "io"
7 "net/http"
8 "sync"
9)
10 
11type Result struct {
12 URL string
13 Size int
14 Err error
15}
16 
17func FetchAll(ctx context.Context, urls []string, maxConcurrent int) []Result {
18 sem := make(chan struct{}, maxConcurrent)
19 results := make([]Result, len(urls))
20 var wg sync.WaitGroup
21 
22 for i, url := range urls {
23 wg.Add(1)
24 go func(idx int, u string) {
25 defer wg.Done()
26 
27 select {
28 case sem <- struct{}{}:
29 defer func() { <-sem }()
30 case <-ctx.Done():
31 results[idx] = Result{URL: u, Err: ctx.Err()}
32 return
33 }
34 
35 results[idx] = fetch(ctx, u)
36 }(i, url)
37 }
38 
39 wg.Wait()
40 return results
41}
42 
43func fetch(ctx context.Context, url string) Result {
44 req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
45 if err != nil {
46 return Result{URL: url, Err: err}
47 }
48 
49 resp, err := http.DefaultClient.Do(req)
50 if err != nil {
51 return Result{URL: url, Err: err}
52 }
53 defer resp.Body.Close()
54 
55 if resp.StatusCode != http.StatusOK {
56 return Result{URL: url, Err: fmt.Errorf("unexpected status %d", resp.StatusCode)}
57 }
58 
59 n, err := io.Copy(io.Discard, resp.Body)
60 return Result{URL: url, Size: int(n), Err: err}
61}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A buffered channel makes a simple counting semaphore that caps concurrent goroutines.
  2. 2Writing each goroutine's output to its own slice index avoids locks on the shared results.
  3. 3Threading a context through both the semaphore acquire and the request lets cancellation short-circuit work cleanly.

Related explainers

java 
public class SlidingLogRateLimiter {
 
    private final int maxRequests;
    private final long windowMillis;

How a sliding-log rate limiter works

rate-limiting concurrency sliding-window
Advanced 8 steps
go 
package theme
 
import (
	"fmt"

Per-tenant HTML templates with Gin's renderer

multi-tenancy concurrency html-templates
Advanced 8 steps
python 
import time
import threading
from enum import Enum
from functools import wraps

Building a circuit breaker in Python

circuit-breaker resilience decorators
Advanced 7 steps
typescript 
import { Injectable } from '@angular/core';
import { HttpInterceptor, HttpRequest, HttpHandler, HttpEvent, HttpErrorResponse } from '@angular/common/http';
import { BehaviorSubject, Observable, throwError } from 'rxjs';
import { catchError, filter, take, switchMap } from 'rxjs/operators';

Refreshing auth tokens in an Angular interceptor

http-interceptor token-refresh rxjs
Advanced 8 steps
go 
package auth
 
import (
	"net/http"

Setting and reading secure session cookies in Go

cookies session-management security
Intermediate 6 steps
php 
<?php
 
namespace App\Http\Middleware;
 

Idempotent requests with Laravel middleware

idempotency middleware caching
Advanced 8 steps

Share this explainer

Here's the card — post it anywhere.

Bounded-concurrency HTTP fetching in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code