Spark Execution Model: Jobs, Stages, Tasks and Shuffles
What actually happens between calling an action and getting a result — and why the shuffle boundary is where most performance problems live.
From code to cluster
Spark builds a lazy logical plan from your transformations. Nothing runs until you call an action (count, collect, write, show). At that point the driver:
- Optimises the logical plan (Catalyst) into a physical plan
- Splits it into a job per action
- Cuts each job into stages at every shuffle boundary
- Runs each stage as a set of tasks — one per partition — on executors
Rendering flowchart
Narrow vs wide transformations
| Narrow (no shuffle) | Wide (shuffle) |
|---|---|
select, filter, withColumn, map | groupBy, join (non-broadcast), distinct, repartition, orderBy |
| Each output partition depends on one input partition | Output partitions depend on many input partitions |
| Pipelined inside one stage | Ends a stage; data is written to disk and fetched over the network |
Watch what a groupBy does to the data — every record travels to the partition that owns its key:
input partition 0
input partition 1
input partition 2
Stage 1 · narrow transformations run in place
A shuffle is expensive because it serialises, writes to local disk, transfers over the network and deserialises every row it moves.
Reading the plan
df = (spark.read.parquet("s3://bucket/orders/")
.filter("status = 'PAID'")
.groupBy("customer_id")
.agg(F.sum("amount").alias("spend")))
df.explain(mode="formatted")Look for Exchange nodes — each one is a shuffle. Exchange hashpartitioning(customer_id, 200) means rows are redistributed into 200 partitions by customer_id.
Practical rules
- Filter and project early so less data reaches the shuffle
- Tune
spark.sql.shuffle.partitions(default 200) to data size; with AQE enabled, Spark can coalesce small shuffle partitions automatically - Prefer broadcast joins when one side is small
- Watch the Spark UI's stage tab: a few tasks taking far longer than the median usually means skew
Finished this note?