Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

PySpark

DataFrame API patterns, schemas, UDF trade-offs and testing Spark code.

5 notes · all topicsData engineering

Start here · beginner2

  1. 1Always Use Explicit Schemas in PySparkSchema inference is convenient in a notebook and dangerous in a pipeline. Here is how to define, enforce and evolve schemas deliberately.
  2. 2PySpark DataFrame Basics in 10 OperationsThe ten DataFrame operations that cover most pipeline code: read, select, filter, withColumn, join, groupBy, window, dedupe, union and write.

Next · intermediate3

  1. 3Broadcast Joins and Handling Data SkewWhen to broadcast, when Spark does it for you, and three techniques for the one partition that takes an hour.
  2. 4Python UDFs vs Built-in Functions vs Pandas UDFsWhy Python UDFs are slow, when a pandas UDF is acceptable, and how to rewrite most UDFs with built-in functions.
  3. 5Testing PySpark Code LocallyStructure transformations as pure DataFrame functions, run them on a local SparkSession in pytest, and compare results reliably.