# Adaptive Query Execution (AQE) in Spark

> How Spark 3 re-plans a query at runtime: coalescing shuffle partitions, switching to broadcast joins and splitting skewed partitions.

- Source: https://akshaythakare.com/notes/apache-spark/adaptive-query-execution
- Topic: Apache Spark
- Level: advanced
- Reading time: 2 min
- Tags: spark, performance, aqe, skew
- Updated: 2026-10-11

## The problem AQE solves

A static plan is chosen before any data is read, using estimates. Estimates are often wrong after filters and joins. AQE re-optimises **between stages**, using the real sizes of shuffle outputs.

```python
spark.conf.set("spark.sql.adaptive.enabled", "true")   # default since Spark 3.2
```

## What it does

| Feature | Without AQE | With AQE |
| --- | --- | --- |
| Coalesce partitions | 200 shuffle partitions, many nearly empty | Merges small partitions into fewer tasks |
| Join strategy | Sort-merge join chosen from estimates | Switches to a broadcast join if one side turns out small |
| Skew join | One partition holds most of a hot key, so one task runs for an hour | Splits the skewed partition into several tasks |

## Settings worth knowing

```python
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128m")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
```

## What AQE does not fix

- Skew in a `groupBy` aggregation on one huge key. Salt the key or pre-aggregate.
- Reading too much data. Partition pruning and column pruning still matter.
- Too many small **input** files. AQE works on shuffles, not on the initial scan.

> **Tip: Interview framing**
>
> "AQE uses real shuffle statistics to merge tiny partitions, switch to broadcast joins and split skewed join partitions. I still design partitioning and check the Spark UI for long-tail tasks."