javascript 36 lines · 8 steps

Uploading records with bounded concurrency

A worker-pool pattern splits records into batches and processes them with a capped number of concurrent uploads.

Explained by highlit
Advanced concurrency worker-pool async-await batching error-handling
1async function uploadInBatches(records, uploadFn, { batchSize = 100, concurrency = 3 } = {}) {
2 const batches = [];
3 for (let i = 0; i < records.length; i += batchSize) {
4 batches.push(records.slice(i, i + batchSize));
5 }
6 
7 const results = [];
8 let cursor = 0;
9 
10 async function worker() {
11 while (cursor < batches.length) {
12 const index = cursor++;
13 const batch = batches[index];
14 try {
15 const response = await uploadFn(batch, index);
16 results[index] = { status: 'fulfilled', count: batch.length, response };
17 } catch (error) {
18 results[index] = { status: 'rejected', count: batch.length, error };
19 }
20 }
21 }
22 
23 const workers = Array.from(
24 { length: Math.min(concurrency, batches.length) },
25 () => worker()
26 );
27 await Promise.all(workers);
28 
29 const failed = results.filter((r) => r.status === 'rejected');
30 return {
31 total: records.length,
32 batches: batches.length,
33 uploaded: results.reduce((sum, r) => sum + (r.status === 'fulfilled' ? r.count : 0), 0),
34 failed,
35 };
36}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A shared cursor lets a fixed set of workers pull the next task without exceeding a concurrency cap.
  2. 2Catching errors per task and recording their status keeps one failure from aborting the whole run.
  3. 3Writing results by index preserves ordering even when tasks finish out of order.

Related explainers

go 
package middleware
 
import (
	"net/http"

Localized validation errors in Gin

middleware internationalization validation
Intermediate 8 steps
java 
package com.example.validation;
 
import java.util.List;
import java.util.stream.Collectors;

Validating JSON payloads against a schema in Java

json-schema validation recursion
Intermediate 8 steps
ruby 
class CircuitBreaker
  class OpenCircuitError < StandardError; end
 
  def initialize(failure_threshold: 5, reset_timeout: 30, half_open_max: 1)

How a circuit breaker guards failing calls

state-machine fault-tolerance concurrency
Advanced 9 steps
javascript 
function formatPhoneNumber(value) {
  const digits = value.replace(/\D/g, '').slice(0, 10);
  const parts = [];
 

Building a live phone number input mask

input-masking regex dom-events
Intermediate 7 steps
go 
func (h *ExportHandler) BulkExport(c *gin.Context) {
	projectID := c.Param("projectID")
 
	reports, err := h.reports.ListByProject(c.Request.Context(), projectID)

Streaming a ZIP download in Gin

streaming zip-archive http-headers
Intermediate 8 steps
javascript 
import { NextResponse } from 'next/server';
 
const locales = ['en', 'fr', 'de', 'es'];
const defaultLocale = 'en';

Locale routing with Next.js middleware

middleware i18n content-negotiation
Intermediate 10 steps

Share this explainer

Here's the card — post it anywhere.

Uploading records with bounded concurrency — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code