# Spark Execution Model: Jobs, Stages, Tasks and Shuffles

> What actually happens between calling an action and getting a result — and why the shuffle boundary is where most performance problems live.

- Source: https://akshaythakare.com/notes/apache-spark/spark-execution-model
- Topic: Apache Spark
- Level: intermediate
- Reading time: 3 min
- Tags: spark, performance, internals
- Updated: 2026-10-11

## From code to cluster

Spark builds a **lazy logical plan** from your transformations. Nothing runs until you call an **action** (`count`, `collect`, `write`, `show`). At that point the driver:

1. Optimises the logical plan (Catalyst) into a physical plan
2. Splits it into a **job** per action
3. Cuts each job into **stages** at every shuffle boundary
4. Runs each stage as a set of **tasks** — one per partition — on executors

```mermaid
flowchart LR
  A[Action] --> J[Job]
  J --> S1[Stage 1<br/>read + filter + map]
  S1 -->|shuffle write / read| S2[Stage 2<br/>groupBy aggregate]
  S2 --> S3[Stage 3<br/>write]
  S1 -.-> T1[task per partition]
```

## Narrow vs wide transformations

| Narrow (no shuffle) | Wide (shuffle) |
| --- | --- |
| `select`, `filter`, `withColumn`, `map` | `groupBy`, `join` (non-broadcast), `distinct`, `repartition`, `orderBy` |
| Each output partition depends on one input partition | Output partitions depend on many input partitions |
| Pipelined inside one stage | Ends a stage; data is written to disk and fetched over the network |

Watch what a `groupBy` does to the data — every record travels to the partition that owns its key:

*(Interactive illustration “Spark Shuffle”: see the web page.)*

A shuffle is expensive because it **serialises, writes to local disk, transfers over the network and deserialises** every row it moves.

## Reading the plan

```python
df = (spark.read.parquet("s3://bucket/orders/")
        .filter("status = 'PAID'")
        .groupBy("customer_id")
        .agg(F.sum("amount").alias("spend")))

df.explain(mode="formatted")
```

Look for `Exchange` nodes — each one is a shuffle. `Exchange hashpartitioning(customer_id, 200)` means rows are redistributed into 200 partitions by `customer_id`.

## Practical rules

- **Filter and project early** so less data reaches the shuffle
- Tune `spark.sql.shuffle.partitions` (default 200) to data size; with AQE enabled, Spark can coalesce small shuffle partitions automatically
- Prefer **broadcast joins** when one side is small
- Watch the Spark UI's **stage** tab: a few tasks taking far longer than the median usually means **skew**

> **Note: Interview angle**
>
> "A stage boundary is a shuffle boundary" is the one-line answer. Follow up with *why* shuffles are expensive and *how* you reduce them.