# Lazy Evaluation: Transformations vs Actions in Spark

> Why nothing runs until you call an action, how Spark builds a plan, and the mistakes that make a job compute the same data twice.

- Source: https://akshaythakare.com/notes/apache-spark/lazy-evaluation
- Topic: Apache Spark
- Level: beginner
- Reading time: 2 min
- Tags: spark, pyspark, fundamentals, interview
- Updated: 2026-10-11

## Two kinds of operations

| Transformations (lazy) | Actions (run the job) |
| --- | --- |
| `select`, `filter`, `withColumn`, `join`, `groupBy`, `union` | `count`, `collect`, `show`, `write`, `take` |
| Return a new DataFrame | Return a result or write data |

```python
df = spark.read.parquet("s3://lake/orders/")          # nothing read yet
paid = df.filter("status = 'paid'").select("store_id", "amount")  # still nothing
paid.write.parquet("s3://lake/paid/")                  # NOW Spark plans and runs
```

## Why lazy?

Spark sees the whole chain before running it, so the optimizer (Catalyst) can:
- push the `filter` down into the Parquet scan, reading fewer rows
- read only the columns you `select` (column pruning)
- combine several steps into one stage

## The double-compute trap

Every action recomputes the plan from the source.

```python
clean = expensive_cleaning(raw)
print(clean.count())                # computes clean
clean.write.parquet(out)            # computes clean AGAIN
```

If a DataFrame is used by more than one action, cache it, and release it afterwards:

```python
clean = expensive_cleaning(raw).cache()
clean.count()        # materialises the cache
clean.write.parquet(out)
clean.unpersist()
```

> **Warning: collect() is not a debugging tool**
>
> `collect()` pulls every row to the driver. On real data it crashes the driver. Use `show(20)` or `take(20)`.

## See the plan

```python
paid.explain(mode="formatted")
```

Look for `PushedFilters` in the scan node. It confirms the filter reached the file reader.