Python Generators for Memory-Safe Data Pipelines
Process files far bigger than RAM by streaming records through small generator stages instead of loading lists.
PythonBeginner3 min readUpdated 11 Oct 2026
The problem with lists
rows = [json.loads(line) for line in open("events.jsonl")] # whole file in memory
valid = [r for r in rows if r.get("user_id")]Each step materialises a full list. A 20 GB file needs well over 20 GB of RAM before anything is written.
Stream instead: one record at a time
A generator yields one value, pauses, and resumes when the next value is requested. Chaining generators gives you a pipeline where only one record is in flight per stage.
Lazy pipeline — one record in flight
read_linesyield line
parseyield dict
validateyield valid row
batchyield 1,000 rows
writebulk insert
import json
from itertools import islice
from typing import Iterator
def read_lines(path: str) -> Iterator[str]:
with open(path, encoding="utf-8") as f:
for line in f:
yield line
def parse(lines: Iterator[str]) -> Iterator[dict]:
for line in lines:
yield json.loads(line)
def valid(rows: Iterator[dict]) -> Iterator[dict]:
for r in rows:
if r.get("user_id") and r.get("ts"):
yield r
def batched(rows: Iterator[dict], size: int = 1000) -> Iterator[list[dict]]:
while chunk := list(islice(rows, size)):
yield chunk
for chunk in batched(valid(parse(read_lines("events.jsonl")))):
insert_many(chunk) # memory stays flat regardless of file sizeWhy it works
- 1
Nothing runs until you iterate
Callingparse(...)builds a generator object; no file is read yet. - 2
The loop pulls one record
for chunk in ...asksbatched, which asksvalid, which asksparse, which asksread_linesfor one line. - 3
Each stage holds one item
Only the current line, dict and the batch buffer exist in memory. - 4
Bulk writes stay efficient
Batching restores throughput for the database while keeping memory bounded.
Finished this note?