Writing Idempotent Airflow DAGs

What "idempotent" means for a task

Running a task for the same logical date twice must leave the target exactly as running it once. Airflow will rerun tasks: retries, backfills, clearing a failed run.

One DAG run Replay

Schedulercreates run for ds

extractds partition only

loadoverwrite partition

publishMERGE by key

notifyafter success

Use the logical date, never "now"

from airflow.decorators import dag, task
import pendulum
 
@dag(schedule="@daily", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=True, max_active_runs=1)
def orders_daily():
 
    @task
    def extract(ds=None):
        # bounded by the run's logical date, not datetime.now()
        return export_orders(start=ds, end=next_day(ds))
 
    @task
    def load(path: str, ds=None):
        # overwrite exactly one partition → rerunning replaces, never appends
        overwrite_partition("s3://lake/orders/", dt=ds, source=path)
 
    load(extract())
 
orders_daily()

Habits that make reruns safe

1. 1

   Bound every read

   Filter by the run's data interval (data_interval_start, data_interval_end).
2. 2

   Overwrite, don't append

   Write to a partition keyed by the logical date and replace it on rerun.
3. 3

   MERGE by natural key

   In the warehouse, upsert instead of INSERT so duplicates cannot accumulate.
4. 4

   Keep side effects last

   Emails and webhooks run after the data step succeeds — and a failed notification never re-triggers the load.