python 35 lines · 8 steps

Splitting a large CSV into smaller files

Stream a big CSV row by row and roll over to a new part file every N rows, repeating the header each time.

Explained by highlit
Intermediate file-io streaming csv chunking pathlib
1import csv
2from pathlib import Path
3 
4 
5def split_csv(source, rows_per_file=100_000, output_dir=None):
6 source = Path(source)
7 output_dir = Path(output_dir) if output_dir else source.parent
8 output_dir.mkdir(parents=True, exist_ok=True)
9 
10 created = []
11 with source.open(newline="", encoding="utf-8") as infile:
12 reader = csv.reader(infile)
13 header = next(reader)
14 
15 chunk_index = 0
16 writer = None
17 outfile = None
18 
19 for row_number, row in enumerate(reader):
20 if row_number % rows_per_file == 0:
21 if outfile is not None:
22 outfile.close()
23 chunk_index += 1
24 target = output_dir / f"{source.stem}_part{chunk_index:03d}.csv"
25 outfile = target.open("w", newline="", encoding="utf-8")
26 writer = csv.writer(outfile)
27 writer.writerow(header)
28 created.append(target)
29 
30 writer.writerow(row)
31 
32 if outfile is not None:
33 outfile.close()
34 
35 return created
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Iterating a reader row by row keeps memory flat no matter how large the source file is.
  2. 2Tracking a single open handle and rolling it over on a boundary is a clean pattern for chunked output.
  3. 3Re-emitting the header into every chunk keeps each output file independently valid.

Related explainers

python 
from flask import Blueprint, jsonify, request, abort
 
v1 = Blueprint("users_v1", __name__)
v2 = Blueprint("users_v2", __name__)

Versioning a Flask API with Blueprints

api versioning blueprints rest
Intermediate 7 steps
php 
<?php
 
namespace App\Http\Controllers;
 

Streaming a filtered CSV export in Laravel

streaming csv-export query-builder
Intermediate 9 steps
python 
import time
import threading
from enum import Enum
from functools import wraps

Building a circuit breaker in Python

circuit-breaker resilience decorators
Advanced 7 steps
python 
from django.contrib.postgres.search import SearchQuery, SearchRank, SearchVector
from django.core.cache import cache
from django.db.models import F
from django.http import JsonResponse

Ranked full-text search in Django

full-text-search debouncing caching
Advanced 9 steps
python 
import os
from datetime import timedelta
 
 

Class-based config in a Flask app factory

configuration app-factory inheritance
Intermediate 7 steps
python 
import time
 
from flask import Flask, g, request
 

Timing requests with Flask hooks

middleware request-lifecycle instrumentation
Intermediate 5 steps

Share this explainer

Here's the card — post it anywhere.

Splitting a large CSV into smaller files — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code