Airflow Backfills and Catchup Without Surprises

Data intervals, not "run time"

Each scheduled DAG run covers a data interval. A daily DAG run for 1 October processes data from 1 October 00:00 to 2 October 00:00, and actually starts after that interval ends.

from airflow.decorators import dag, task
import pendulum
 
@dag(schedule="@daily", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=False)
def daily_sales():
    @task
    def load(data_interval_start=None, data_interval_end=None):
        # process exactly this window — never "now() minus one day"
        run_extract(start=data_interval_start, end=data_interval_end)
    load()
 
daily_sales()

catchup

• catchup=True: when a DAG is enabled (or resumed after a pause), Airflow creates a run for every missed interval since start_date.
• catchup=False: only the most recent interval is scheduled.

Turn catchup on only when every missed interval genuinely needs processing — otherwise enabling a DAG with an old start_date can launch hundreds of runs.

Backfills

A backfill reruns a DAG for a chosen historical range, for example after fixing a bug:

# Airflow 3
airflow backfill create --dag-id daily_sales --from-date 2026-09-01 --to-date 2026-09-30
 
# Airflow 2
airflow dags backfill daily_sales --start-date 2026-09-01 --end-date 2026-09-30

Limit parallel load on sources and warehouses with max_active_runs on the DAG and pools or concurrency limits on heavy tasks.

Make reruns safe

A backfill is only safe if each run is idempotent for its interval:

• Overwrite the interval's partition (INSERT OVERWRITE ... PARTITION (order_date = ...)) or MERGE on a key — never blindly append.
• Read inputs for the interval only.
• Keep side effects (emails, API calls) out of reprocessing paths or guard them.

Checklist before a big backfill

1. Confirm the DAG uses data-interval variables everywhere.
2. Confirm writes overwrite or merge per interval.
3. Set max_active_runs and pools to protect downstream systems.
4. Run one interval first and compare row counts with the original.