# Writing Idempotent Airflow DAGs

> Logical dates, partition overwrites and safe retries — the habits that make re-running a DAG boring.

- Source: https://akshaythakare.com/notes/apache-airflow/idempotent-dags
- Topic: Apache Airflow
- Level: intermediate
- Reading time: 2 min
- Tags: airflow, orchestration, idempotency
- Updated: 2026-10-11

## What "idempotent" means for a task

Running a task for the same **logical date** twice must leave the target exactly as running it once. Airflow *will* rerun tasks: retries, backfills, clearing a failed run.

**One DAG run:** Scheduler (creates run for ds) → extract (ds partition only) → load (overwrite partition) → publish (MERGE by key) → notify (after success)

## Use the logical date, never "now"

```python
from airflow.decorators import dag, task
import pendulum

@dag(schedule="@daily", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), catchup=True, max_active_runs=1)
def orders_daily():

    @task
    def extract(ds=None):
        # bounded by the run's logical date, not datetime.now()
        return export_orders(start=ds, end=next_day(ds))

    @task
    def load(path: str, ds=None):
        # overwrite exactly one partition → rerunning replaces, never appends
        overwrite_partition("s3://lake/orders/", dt=ds, source=path)

    load(extract())

orders_daily()
```

## Habits that make reruns safe

  
### Bound every read
Filter by the run's data interval (`data_interval_start`, `data_interval_end`).
  
### Overwrite, don't append
Write to a partition keyed by the logical date and replace it on rerun.
  
### MERGE by natural key
In the warehouse, upsert instead of `INSERT` so duplicates cannot accumulate.
  
### Keep side effects last
Emails and webhooks run after the data step succeeds — and a failed notification never re-triggers the load.

> **Note: Backfills become trivial**
>
> When every task is bounded and idempotent, `airflow dags backfill -s 2026-01-01 -e 2026-01-31 orders_daily` is safe to run as often as you need.