Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

Lazy Evaluation: Transformations vs Actions in Spark

Why nothing runs until you call an action, how Spark builds a plan, and the mistakes that make a job compute the same data twice.

Apache SparkBeginner2 min readUpdated 11 Oct 2026

Two kinds of operations

Transformations (lazy)Actions (run the job)
select, filter, withColumn, join, groupBy, unioncount, collect, show, write, take
Return a new DataFrameReturn a result or write data
Python.py
df = spark.read.parquet("s3://lake/orders/")          # nothing read yet
paid = df.filter("status = 'paid'").select("store_id", "amount")  # still nothing
paid.write.parquet("s3://lake/paid/")                  # NOW Spark plans and runs

Why lazy?

Spark sees the whole chain before running it, so the optimizer (Catalyst) can:

  • push the filter down into the Parquet scan, reading fewer rows
  • read only the columns you select (column pruning)
  • combine several steps into one stage

The double-compute trap

Every action recomputes the plan from the source.

Python.py
clean = expensive_cleaning(raw)
print(clean.count())                # computes clean
clean.write.parquet(out)            # computes clean AGAIN

If a DataFrame is used by more than one action, cache it, and release it afterwards:

Python.py
clean = expensive_cleaning(raw).cache()
clean.count()        # materialises the cache
clean.write.parquet(out)
clean.unpersist()

See the plan

Python.py
paid.explain(mode="formatted")

Look for PushedFilters in the scan node. It confirms the filter reached the file reader.

Finished this note?

Related notes

Share

LinkedInXWhatsApp