python 35 lines · 8 steps

Splitting a large CSV into smaller files

Stream a big CSV row by row and roll over to a new part file every N rows, repeating the header each time.

Explained by highlit
Intermediate file-io streaming csv chunking pathlib
1import csv
2from pathlib import Path
3 
4 
5def split_csv(source, rows_per_file=100_000, output_dir=None):
6 source = Path(source)
7 output_dir = Path(output_dir) if output_dir else source.parent
8 output_dir.mkdir(parents=True, exist_ok=True)
9 
10 created = []
11 with source.open(newline="", encoding="utf-8") as infile:
12 reader = csv.reader(infile)
13 header = next(reader)
14 
15 chunk_index = 0
16 writer = None
17 outfile = None
18 
19 for row_number, row in enumerate(reader):
20 if row_number % rows_per_file == 0:
21 if outfile is not None:
22 outfile.close()
23 chunk_index += 1
24 target = output_dir / f"{source.stem}_part{chunk_index:03d}.csv"
25 outfile = target.open("w", newline="", encoding="utf-8")
26 writer = csv.writer(outfile)
27 writer.writerow(header)
28 created.append(target)
29 
30 writer.writerow(row)
31 
32 if outfile is not None:
33 outfile.close()
34 
35 return created
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Iterating a reader row by row keeps memory flat no matter how large the source file is.
  2. 2Tracking a single open handle and rolling it over on a boundary is a clean pattern for chunked output.
  3. 3Re-emitting the header into every chunk keeps each output file independently valid.

Related explainers

python 
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
 
app = FastAPI()
 

Building a WebSocket chat with FastAPI

websockets broadcast connection-management
Intermediate 9 steps
python 
import time
import uuid
 
from django.utils.deprecation import MiddlewareMixin

Attaching per-request context in Django

middleware request lifecycle multi-tenancy
Intermediate 7 steps
python 
import random
from typing import Iterator, List
 
 

How reservoir sampling picks k items

reservoir-sampling streaming randomness
Intermediate 5 steps
python 
import secrets
 
from django.contrib.auth import authenticate, login
from django.core.cache import cache

Two-factor login with OTP in Django

two-factor-auth one-time-passwords caching
Intermediate 9 steps
python 
import re
from functools import total_ordering
from typing import Optional
 

Parsing and comparing semantic versions

regex operator-overloading sorting
Intermediate 7 steps
python 
from typing import Any, Sequence, Mapping
 
 
def render_markdown_table(

Rendering an aligned Markdown table in Python

string formatting data transformation closures
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Splitting a large CSV into smaller files — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code