go 61 lines · 9 steps

Bounded-concurrency HTTP fetching in Go

Fetch many URLs in parallel while capping in-flight requests with a semaphore channel, honoring context cancellation.

Explained by highlit
Advanced concurrency semaphore context-cancellation goroutines waitgroup
1package fetcher
2 
3import (
4 "context"
5 "fmt"
6 "io"
7 "net/http"
8 "sync"
9)
10 
11type Result struct {
12 URL string
13 Size int
14 Err error
15}
16 
17func FetchAll(ctx context.Context, urls []string, maxConcurrent int) []Result {
18 sem := make(chan struct{}, maxConcurrent)
19 results := make([]Result, len(urls))
20 var wg sync.WaitGroup
21 
22 for i, url := range urls {
23 wg.Add(1)
24 go func(idx int, u string) {
25 defer wg.Done()
26 
27 select {
28 case sem <- struct{}{}:
29 defer func() { <-sem }()
30 case <-ctx.Done():
31 results[idx] = Result{URL: u, Err: ctx.Err()}
32 return
33 }
34 
35 results[idx] = fetch(ctx, u)
36 }(i, url)
37 }
38 
39 wg.Wait()
40 return results
41}
42 
43func fetch(ctx context.Context, url string) Result {
44 req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
45 if err != nil {
46 return Result{URL: url, Err: err}
47 }
48 
49 resp, err := http.DefaultClient.Do(req)
50 if err != nil {
51 return Result{URL: url, Err: err}
52 }
53 defer resp.Body.Close()
54 
55 if resp.StatusCode != http.StatusOK {
56 return Result{URL: url, Err: fmt.Errorf("unexpected status %d", resp.StatusCode)}
57 }
58 
59 n, err := io.Copy(io.Discard, resp.Body)
60 return Result{URL: url, Size: int(n), Err: err}
61}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A buffered channel makes a simple counting semaphore that caps concurrent goroutines.
  2. 2Writing each goroutine's output to its own slice index avoids locks on the shared results.
  3. 3Threading a context through both the semaphore acquire and the request lets cancellation short-circuit work cleanly.

Related explainers

go 
package streaming
 
import (
	"bufio"

Streaming NDJSON logs over HTTP in Go

http-streaming channels select
Advanced 10 steps
go 
package api
 
import (
	"crypto/sha256"

ETag conditional requests in Gin

http-caching etag conditional-requests
Intermediate 6 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps
rust 
use std::collections::VecDeque;
use std::sync::{Arc, Condvar, Mutex};
use std::time::{Duration, Instant};
 

Building a counting semaphore in Rust

concurrency synchronization condition-variable
Advanced 9 steps
go 
package middleware
 
import (
	"net/http"

Per-plan export limits in Gin middleware

middleware rate-limiting authorization
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Bounded-concurrency HTTP fetching in Go — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code