Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

Spark Memory Management in Practice

Executor memory regions, why unified memory lets execution evict cache, and how to reason about out-of-memory errors instead of just adding RAM.

Apache SparkAdvanced2 min readUpdated 11 Oct 2026

The executor's memory layout

An executor JVM heap (spark.executor.memory) is divided roughly like this:

Plain text.txt
┌──────────────────────────────── executor heap ────────────────────────────────┐
│ reserved (300 MB) │ user memory (1 − fraction) │ unified Spark memory (fraction) │
│                   │  your objects, UDF data     │  execution  ⇄  storage          │
└───────────────────────────────────────────────────────────────────────────────┘
  • spark.memory.fraction (default 0.6) — share of heap (after reserved) for unified memory
  • spark.memory.storageFraction (default 0.5) — part of unified memory protected for cached data
  • Off-heap / overhead (spark.executor.memoryOverhead) — native memory for Python workers, network buffers and off-heap structures. PySpark UDFs live here, not in the JVM heap.

Unified memory: execution can evict storage

Execution memory (shuffles, joins, sorts, aggregations) and storage memory (cache) share one pool. Execution can evict cached blocks down to the protected storage region; storage cannot evict execution. That is why caching a large DataFrame can silently slow a job: cached blocks get evicted and recomputed.

Common OOM causes and fixes

SymptomLikely causeFix
Driver OOMcollect(), toPandas(), huge broadcastWrite results out; cap broadcast size
Executor OOM in one taskSkewed key — one partition far largerSalt the key, enable AQE skew join, pre-aggregate
Container killed by YARN/K8sOverhead exceeded (Python UDFs, Arrow)Raise memoryOverhead; prefer built-in functions to Python UDFs
Spill to disk, slow stagesToo few shuffle partitionsIncrease partitions or let AQE coalesce
Python.py
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

A diagnostic habit

Before adding memory, open the Spark UI and answer: which stage, which task, how much shuffle read, and is one task an outlier? Most "memory problems" are data-distribution problems.

Finished this note?

Related notes

Share

LinkedInXWhatsApp