ruby 32 lines · 5 steps

Batching bulk inserts in Rails

A BulkImporter chunks records into fixed-size batches and inserts each with a single deduplicated SQL statement.

Explained by highlit
Rails Intermediate batching bulk-insert deduplication data-normalization
1class BulkImporter
2 BATCH_SIZE = 500
3 
4 def initialize(records)
5 @records = records
6 end
7 
8 def import!
9 inserted_ids = []
10 
11 @records.each_slice(BATCH_SIZE) do |batch|
12 rows = batch.map do |record|
13 {
14 email: record.fetch(:email).downcase.strip,
15 name: record[:name],
16 created_at: Time.current,
17 updated_at: Time.current
18 }
19 end
20 
21 result = User.insert_all(
22 rows,
23 unique_by: :email,
24 returning: %w[id]
25 )
26 
27 inserted_ids.concat(result.rows.flatten)
28 end
29 
30 inserted_ids
31 end
32end
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Slicing records into fixed batches keeps memory and SQL statement size bounded on large imports.
  2. 2insert_all bypasses model callbacks, so timestamps and normalization must be set manually.
  3. 3unique_by lets the database skip duplicates atomically instead of checking in Ruby.

Related explainers

ruby 
class Credentials
  SENSITIVE = %i[password api_key secret_token access_key].freeze
  REDACTED = "[REDACTED]".freeze
 

Redacting secrets in Ruby object output

serialization introspection security
Intermediate 7 steps
java 
public class OrderRepository {
 
    private static final int BATCH_SIZE = 500;
    private static final String INSERT_SQL =

Batched JDBC inserts with transaction safety

jdbc batching transactions
Intermediate 8 steps
typescript 
type Fetcher<T> = (key: string) => Promise<T>;
 
export class RequestDeduplicator<T> {
  private inFlight = new Map<string, Promise<T>>();

Deduplicating in-flight requests in TypeScript

deduplication promises caching
Intermediate 7 steps
ruby 
module Multitenant
  extend ActiveSupport::Concern
 
  included do

How a multitenant concern scopes Rails models

multitenancy default-scope concern
Advanced 8 steps
ruby 
module Api
  module V1
    class BaseController < ActionController::API
      rescue_from StandardError, with: :render_internal_error

Centralized API error handling in Rails

error-handling rescue-from json-api
Intermediate 7 steps
ruby 
require "openssl"
require "base64"
 
module WebhookSignature

Signing and verifying webhooks in Ruby

hmac cryptography webhooks
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Batching bulk inserts in Rails — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code