# Airflow Backfills and Catchup Without Surprises

> How data intervals, catchup and backfills work in Airflow, and how to design DAGs so rerunning last month is safe and boring.

- Source: https://akshaythakare.com/notes/apache-airflow/backfills-and-catchup
- Topic: Apache Airflow
- Level: intermediate
- Reading time: 3 min
- Tags: airflow, orchestration, backfill, idempotency
- Updated: 2026-10-11

## Data intervals, not "run time"

Each scheduled DAG run covers a **data interval**. A daily DAG run for 1 October processes data from 1 October 00:00 to 2 October 00:00, and actually starts *after* that interval ends.

```python
from airflow.decorators import dag, task
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=False)
def daily_sales():
    @task
    def load(data_interval_start=None, data_interval_end=None):
        # process exactly this window — never "now() minus one day"
        run_extract(start=data_interval_start, end=data_interval_end)
    load()

daily_sales()
```

> **Warning: Never use the wall clock inside a task**
>
> `datetime.now()` makes a rerun process different data from the original run. Use the run's `data_interval_start` and `data_interval_end` so every run is reproducible.

## catchup

- `catchup=True`: when a DAG is enabled (or resumed after a pause), Airflow creates a run for every missed interval since `start_date`.
- `catchup=False`: only the most recent interval is scheduled.

Turn catchup on only when every missed interval genuinely needs processing — otherwise enabling a DAG with an old `start_date` can launch hundreds of runs.

## Backfills

A backfill reruns a DAG for a chosen historical range, for example after fixing a bug:

```bash
# Airflow 3
airflow backfill create --dag-id daily_sales --from-date 2026-09-01 --to-date 2026-09-30

# Airflow 2
airflow dags backfill daily_sales --start-date 2026-09-01 --end-date 2026-09-30
```

Limit parallel load on sources and warehouses with `max_active_runs` on the DAG and pools or concurrency limits on heavy tasks.

## Make reruns safe

A backfill is only safe if each run is **idempotent** for its interval:

- Overwrite the interval's partition (`INSERT OVERWRITE ... PARTITION (order_date = ...)`) or `MERGE` on a key — never blindly append.
- Read inputs for the interval only.
- Keep side effects (emails, API calls) out of reprocessing paths or guard them.

```mermaid
flowchart TB
  accTitle: Idempotent interval processing
  accDescr: Each run reads one interval, writes into that interval's partition with overwrite or merge, so rerunning replaces rather than duplicates.
  I[Interval 2026-09-14] --> X[Extract window] --> W[Overwrite partition 2026-09-14]
```

## Checklist before a big backfill

1. Confirm the DAG uses data-interval variables everywhere.
2. Confirm writes overwrite or merge per interval.
3. Set `max_active_runs` and pools to protect downstream systems.
4. Run one interval first and compare row counts with the original.