# Batch vs Streaming: Choosing the Right Pipeline

> Latency requirements, correctness, cost and operations — a practical framework for the most common data system design question.

- Source: https://akshaythakare.com/notes/system-design/batch-vs-streaming
- Topic: Data Engineering System Design
- Level: intermediate
- Reading time: 3 min
- Tags: system-design, streaming, batch, kafka, interview
- Updated: 2026-10-11

## Start from the requirement, not the tool

Ask: **how stale can the answer be before it loses value?** If the honest answer is "hours", streaming adds cost and operational risk for no benefit.

## Two shapes

**Batch:** Sources (daily snapshot) → Extract (bounded window) → Lake (immutable files) → Warehouse (one transaction) → Dashboard (refreshed hourly)

**Streaming:** Events (continuous) → Stream job (windows + state) → Sink (upserts) → Serving (seconds fresh) → Alerts (real time)

## Comparison

| Concern | Batch | Streaming |
| --- | --- | --- |
| Freshness | Minutes to hours | Seconds |
| Correctness | Easy to reason about; replay a window | Needs event time, watermarks, late-data handling |
| Cost | Compute only while running | Always-on clusters and brokers |
| Operations | Rerun the job | Offsets, state stores, back-pressure, schema evolution |
| Typical use | Finance reporting, ML features, daily KPIs | Fraud detection, IoT monitoring, live personalisation |

## A decision path

  
### Define the freshness SLA
Write it down in minutes. "Real time" is not a number.
  
### Check the consumers
A dashboard refreshed hourly gains nothing from second-level data.
  
### Consider micro-batch
Running a batch job every 5–15 minutes often meets the SLA with batch simplicity.
  
### Go streaming when the action is immediate
Blocking a fraudulent payment or reacting to a sensor needs true streaming.

> **Tip: Interview framing**
>
> State the requirement, pick the simplest design that meets it, then name what would make you change your mind — "if the SLA dropped below a minute, I'd move ingestion to Kafka and Structured Streaming".