Adaptive Query Execution (AQE) in Spark
How Spark 3 re-plans a query at runtime: coalescing shuffle partitions, switching to broadcast joins and splitting skewed partitions.
Apache SparkAdvanced2 min readUpdated 11 Oct 2026
The problem AQE solves
A static plan is chosen before any data is read, using estimates. Estimates are often wrong after filters and joins. AQE re-optimises between stages, using the real sizes of shuffle outputs.
spark.conf.set("spark.sql.adaptive.enabled", "true") # default since Spark 3.2What it does
| Feature | Without AQE | With AQE |
|---|---|---|
| Coalesce partitions | 200 shuffle partitions, many nearly empty | Merges small partitions into fewer tasks |
| Join strategy | Sort-merge join chosen from estimates | Switches to a broadcast join if one side turns out small |
| Skew join | One partition holds most of a hot key, so one task runs for an hour | Splits the skewed partition into several tasks |
Settings worth knowing
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128m")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")What AQE does not fix
- Skew in a
groupByaggregation on one huge key. Salt the key or pre-aggregate. - Reading too much data. Partition pruning and column pruning still matter.
- Too many small input files. AQE works on shuffles, not on the initial scan.
Finished this note?