Data Pipeline Observability: Freshness, Volume and Quality

A green job can still produce wrong data

Job success means the code finished. It does not mean the data is complete, fresh or correct. Data observability watches the data itself.

The five signals

Signal	Question	Example check
Freshness	Is the latest data recent enough?	max(loaded_at) > now() - 2h
Volume	Did we get the expected number of rows?	Today's rows within ±30% of the 4-week average for this weekday
Schema	Did a column appear, disappear or change type?	Compare to the declared contract
Quality	Are values valid?	Nulls, uniqueness, ranges, referential integrity
Lineage	What breaks downstream if this fails?	Table → dashboard dependency graph

Reconcile with the source

The strongest check compares your totals to an independent number from the source system, such as end-of-day transaction counts. A match proves completeness; a mismatch tells you how much is missing.

Alert design

• Alert on symptoms users feel (stale dashboard, wrong totals), not on every warning.
• Route each alert to an owner, with a runbook link.
• Separate "page now" (data wrong or very late) from "review tomorrow" (minor drift).