# Python Generators for Memory-Safe Data Pipelines

> Process files far bigger than RAM by streaming records through small generator stages instead of loading lists.

- Source: https://akshaythakare.com/notes/python/generators-for-data-pipelines
- Topic: Python
- Level: beginner
- Reading time: 3 min
- Tags: python, generators, memory, etl
- Updated: 2026-10-11

## The problem with lists

```python
rows = [json.loads(line) for line in open("events.jsonl")]   # whole file in memory
valid = [r for r in rows if r.get("user_id")]
```

Each step materialises a full list. A 20 GB file needs well over 20 GB of RAM before anything is written.

## Stream instead: one record at a time

A **generator** yields one value, pauses, and resumes when the next value is requested. Chaining generators gives you a pipeline where only one record is in flight per stage.

**Lazy pipeline — one record in flight:** read_lines (yield line) → parse (yield dict) → validate (yield valid row) → batch (yield 1,000 rows) → write (bulk insert)

```python
import json
from itertools import islice
from typing import Iterator

def read_lines(path: str) -> Iterator[str]:
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line

def parse(lines: Iterator[str]) -> Iterator[dict]:
    for line in lines:
        yield json.loads(line)

def valid(rows: Iterator[dict]) -> Iterator[dict]:
    for r in rows:
        if r.get("user_id") and r.get("ts"):
            yield r

def batched(rows: Iterator[dict], size: int = 1000) -> Iterator[list[dict]]:
    while chunk := list(islice(rows, size)):
        yield chunk

for chunk in batched(valid(parse(read_lines("events.jsonl")))):
    insert_many(chunk)          # memory stays flat regardless of file size
```

## Why it works

  
### Nothing runs until you iterate
Calling `parse(...)` builds a generator object; no file is read yet.
  
### The loop pulls one record
`for chunk in ...` asks `batched`, which asks `valid`, which asks `parse`, which asks `read_lines` for one line.
  
### Each stage holds one item
Only the current line, dict and the batch buffer exist in memory.
  
### Bulk writes stay efficient
Batching restores throughput for the database while keeping memory bounded.

> **Tip: Rule of thumb**
>
> Use generators for *streaming* transformations. Reach for pandas, Polars or Spark when you need joins, sorts or aggregations across the whole dataset.