javascript 36 lines · 8 steps

Uploading records with bounded concurrency

A worker-pool pattern splits records into batches and processes them with a capped number of concurrent uploads.

Explained by highlit
Advanced concurrency worker-pool async-await batching error-handling
1async function uploadInBatches(records, uploadFn, { batchSize = 100, concurrency = 3 } = {}) {
2 const batches = [];
3 for (let i = 0; i < records.length; i += batchSize) {
4 batches.push(records.slice(i, i + batchSize));
5 }
6 
7 const results = [];
8 let cursor = 0;
9 
10 async function worker() {
11 while (cursor < batches.length) {
12 const index = cursor++;
13 const batch = batches[index];
14 try {
15 const response = await uploadFn(batch, index);
16 results[index] = { status: 'fulfilled', count: batch.length, response };
17 } catch (error) {
18 results[index] = { status: 'rejected', count: batch.length, error };
19 }
20 }
21 }
22 
23 const workers = Array.from(
24 { length: Math.min(concurrency, batches.length) },
25 () => worker()
26 );
27 await Promise.all(workers);
28 
29 const failed = results.filter((r) => r.status === 'rejected');
30 return {
31 total: records.length,
32 batches: batches.length,
33 uploaded: results.reduce((sum, r) => sum + (r.status === 'fulfilled' ? r.count : 0), 0),
34 failed,
35 };
36}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A shared cursor lets a fixed set of workers pull the next task without exceeding a concurrency cap.
  2. 2Catching errors per task and recording their status keeps one failure from aborting the whole run.
  3. 3Writing results by index preserves ordering even when tasks finish out of order.

Related explainers

rust 
use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json
Intermediate 6 steps
ruby 
require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection
Intermediate 7 steps
javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
javascript 
import { useState, useEffect, useCallback, useRef } from 'react';
 
const cache = new Map();
const inflight = new Map();

Building a stale-while-revalidate hook in React

caching request-deduplication custom-hooks
Advanced 10 steps
go 
func (w *Watcher) resetDebounce(d time.Duration) {
	if !w.timer.Stop() {
		select {
		case <-w.timer.C:

Debouncing a stream of events in Go

debounce timers channels
Advanced 7 steps
rust 
use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Uploading records with bounded concurrency — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code