# Spark Memory Management in Practice

> Executor memory regions, why unified memory lets execution evict cache, and how to reason about out-of-memory errors instead of just adding RAM.

- Source: https://akshaythakare.com/notes/apache-spark/spark-memory-management
- Topic: Apache Spark
- Level: advanced
- Reading time: 2 min
- Tags: spark, memory, tuning
- Updated: 2026-10-11

## The executor's memory layout

An executor JVM heap (`spark.executor.memory`) is divided roughly like this:

```text
┌──────────────────────────────── executor heap ────────────────────────────────┐
│ reserved (300 MB) │ user memory (1 − fraction) │ unified Spark memory (fraction) │
│                   │  your objects, UDF data     │  execution  ⇄  storage          │
└───────────────────────────────────────────────────────────────────────────────┘
```

- `spark.memory.fraction` (default 0.6) — share of heap (after reserved) for **unified memory**
- `spark.memory.storageFraction` (default 0.5) — part of unified memory protected for **cached data**
- **Off-heap / overhead** (`spark.executor.memoryOverhead`) — native memory for Python workers, network buffers and off-heap structures. PySpark UDFs live here, not in the JVM heap.

## Unified memory: execution can evict storage

Execution memory (shuffles, joins, sorts, aggregations) and storage memory (cache) share one pool. Execution can **evict cached blocks** down to the protected storage region; storage cannot evict execution. That is why caching a large DataFrame can silently slow a job: cached blocks get evicted and recomputed.

## Common OOM causes and fixes

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Driver OOM | `collect()`, `toPandas()`, huge broadcast | Write results out; cap broadcast size |
| Executor OOM in one task | **Skewed key** — one partition far larger | Salt the key, enable AQE skew join, pre-aggregate |
| Container killed by YARN/K8s | Overhead exceeded (Python UDFs, Arrow) | Raise `memoryOverhead`; prefer built-in functions to Python UDFs |
| Spill to disk, slow stages | Too few shuffle partitions | Increase partitions or let AQE coalesce |

```python
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
```

## A diagnostic habit

Before adding memory, open the Spark UI and answer: **which stage**, **which task**, **how much shuffle read**, and **is one task an outlier**? Most "memory problems" are data-distribution problems.