# Step Functions vs Airflow — and Using Both

> Airflow is a scheduler and dependency manager; Step Functions is a durable AWS-native state machine. Here is how to split responsibilities.

- Source: https://akshaythakare.com/notes/aws/step-functions-vs-airflow
- Topic: AWS
- Level: intermediate
- Reading time: 3 min
- Tags: aws, step-functions, airflow, orchestration
- Updated: 2026-10-11

## What each is good at

| | Apache Airflow | AWS Step Functions |
| --- | --- | --- |
| Model | DAGs of Python-defined tasks | JSON/YAML state machines (ASL) |
| Strength | Scheduling, backfills, cross-system dependencies, rich operator ecosystem | Durable, serverless coordination of AWS services with built-in retries and `.sync` integrations |
| Runs where | Your scheduler + workers (or MWAA / Astronomer) | Fully managed |
| State | Metadata database | Execution history (Standard workflows keep up to 90 days) |
| Cost model | Infrastructure | Per state transition (Standard) |

## A clean split

**Who does what:** Airflow (decides when) → Step Functions (coordinates AWS) → Glue (extract + transform) → Lambda (validate + lock) → Snowflake (publish)

Let **Airflow decide when**, and let **Step Functions decide how** inside AWS:

```mermaid
sequenceDiagram
  participant AF as Airflow DAG
  participant SF as Step Functions
  participant G as Glue / Lambda
  AF->>SF: StartExecution(batch_id, config)
  SF->>G: Lambda acquire lock
  SF->>G: glue:startJobRun.sync (extract)
  SF->>G: Lambda validate manifest
  SF->>G: glue:startJobRun.sync (transform, publish)
  SF->>G: Lambda release lock
  AF->>SF: sensor polls DescribeExecution
  SF-->>AF: SUCCEEDED / FAILED
```

```python
start = StepFunctionStartExecutionOperator(
    task_id="start", state_machine_arn=STATE_MACHINE_ARN,
    state_machine_input={"batch_id": "{{ ds_nodash }}"})
wait = StepFunctionExecutionSensor(
    task_id="wait", execution_arn=start.output, mode="reschedule", poke_interval=30)
start >> wait
```

## Rules that keep it safe

- Pass **IDs and configuration**, never data rows, through orchestration state
- Make every task **idempotent** — retries will happen
- `max_active_runs=1` only protects runs started by that DAG; use a real lock if anything else can start the workflow
- Notification failures must never trigger a republish