Batch vs Streaming: Choosing the Right Pipeline

Latency requirements, correctness, cost and operations — a practical framework for the most common data system design question.

Data Engineering System DesignIntermediate3 min readUpdated 11 Oct 2026

~/notes/system-design/batch-vs-streaming

Start from the requirement, not the tool

Ask: how stale can the answer be before it loses value? If the honest answer is "hours", streaming adds cost and operational risk for no benefit.

Batch

Sourcesdaily snapshot

Extractbounded window

Lakeimmutable files

Warehouseone transaction

Dashboardrefreshed hourly

Streaming

Eventscontinuous

Stream jobwindows + state

Sinkupserts

Servingseconds fresh

Alertsreal time

Concern	Batch	Streaming
Freshness	Minutes to hours	Seconds
Correctness	Easy to reason about; replay a window	Needs event time, watermarks, late-data handling
Cost	Compute only while running	Always-on clusters and brokers
Operations	Rerun the job	Offsets, state stores, back-pressure, schema evolution
Typical use	Finance reporting, ML features, daily KPIs	Fraud detection, IoT monitoring, live personalisation

1
Define the freshness SLA
Write it down in minutes. "Real time" is not a number.
2
Check the consumers
A dashboard refreshed hourly gains nothing from second-level data.
3
Consider micro-batch
Running a batch job every 5–15 minutes often meets the SLA with batch simplicity.
4
Go streaming when the action is immediate
Blocking a fraudulent payment or reacting to a sensor needs true streaming.

Finished this note?