# Data Pipeline Observability: Freshness, Volume and Quality

> What to monitor so you find broken data before your users do: freshness, volume, schema, quality checks, lineage and alert design.

- Source: https://akshaythakare.com/notes/system-design/pipeline-observability
- Topic: Data Engineering System Design
- Level: intermediate
- Reading time: 2 min
- Tags: system-design, monitoring, data-quality, observability
- Updated: 2026-10-11

## A green job can still produce wrong data

Job success means the code finished. It does not mean the data is complete, fresh or correct. Data observability watches the **data itself**.

## The five signals

| Signal | Question | Example check |
| --- | --- | --- |
| **Freshness** | Is the latest data recent enough? | `max(loaded_at) > now() - 2h` |
| **Volume** | Did we get the expected number of rows? | Today's rows within ±30% of the 4-week average for this weekday |
| **Schema** | Did a column appear, disappear or change type? | Compare to the declared contract |
| **Quality** | Are values valid? | Nulls, uniqueness, ranges, referential integrity |
| **Lineage** | What breaks downstream if this fails? | Table → dashboard dependency graph |

## Reconcile with the source

The strongest check compares your totals to an independent number from the source system, such as end-of-day transaction counts. A match proves completeness; a mismatch tells you how much is missing.

## Alert design

- Alert on **symptoms users feel** (stale dashboard, wrong totals), not on every warning.
- Route each alert to an owner, with a runbook link.
- Separate "page now" (data wrong or very late) from "review tomorrow" (minor drift).

> **Tip: Interview framing**
>
> "Besides job failures, I monitor freshness, volume against the source, schema and key quality checks, and I reconcile totals with an independent source number."