Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

Apache Spark

Execution model, memory, shuffles, partitioning and performance tuning.

5 notes · all topicsData engineering

Start here · beginner1

  1. 1Lazy Evaluation: Transformations vs Actions in SparkWhy nothing runs until you call an action, how Spark builds a plan, and the mistakes that make a job compute the same data twice.

Next · intermediate2

  1. 2Spark Partitions and File Sizes: repartition, coalesce and the Small-Files ProblemHow in-memory partitions become output files, when to repartition or coalesce, and how to stop a job from writing ten thousand tiny files.
  2. 3Spark Execution Model: Jobs, Stages, Tasks and ShufflesWhat actually happens between calling an action and getting a result — and why the shuffle boundary is where most performance problems live.

Go deeper · advanced2

  1. 4Adaptive Query Execution (AQE) in SparkHow Spark 3 re-plans a query at runtime: coalescing shuffle partitions, switching to broadcast joins and splitting skewed partitions.
  2. 5Spark Memory Management in PracticeExecutor memory regions, why unified memory lets execution evict cache, and how to reason about out-of-memory errors instead of just adding RAM.