# PySpark DataFrame Basics in 10 Operations

> The ten DataFrame operations that cover most pipeline code: read, select, filter, withColumn, join, groupBy, window, dedupe, union and write.

- Source: https://akshaythakare.com/notes/pyspark/dataframe-basics
- Topic: PySpark
- Level: beginner
- Reading time: 2 min
- Tags: pyspark, spark, fundamentals
- Updated: 2026-10-11

```python
from pyspark.sql import SparkSession, functions as F, Window

spark = SparkSession.builder.appName("orders").getOrCreate()
```

## 1–3. Read, select, filter

```python
orders = spark.read.parquet("s3://lake/raw/orders/")
paid = orders.select("order_id", "store_id", "amount", "status", "ts").filter(F.col("status") == "paid")
```

## 4. Add or change a column

```python
paid = paid.withColumn("business_date", F.to_date("ts"))
```

## 5. Join

```python
stores = spark.read.parquet("s3://lake/dim/stores/")
enriched = paid.join(stores, "store_id", "left")
```

## 6. Aggregate

```python
daily = enriched.groupBy("business_date", "region").agg(
    F.sum("amount").alias("net_sales"),
    F.countDistinct("order_id").alias("orders"),
)
```

## 7. Window

```python
w = Window.partitionBy("region").orderBy("business_date")
daily = daily.withColumn("running_sales", F.sum("net_sales").over(w))
```

## 8. Deduplicate (latest per key)

```python
w = Window.partitionBy("order_id").orderBy(F.col("ts").desc())
latest = orders.withColumn("rn", F.row_number().over(w)).filter("rn = 1").drop("rn")
```

`dropDuplicates(["order_id"])` keeps an arbitrary row. Use the window when "latest" matters.

## 9. Union

```python
both = jan.unionByName(feb, allowMissingColumns=True)   # by name, not position
```

## 10. Write

```python
daily.write.mode("overwrite").partitionBy("business_date").parquet("s3://lake/curated/daily/")
```

> **Tip: Prefer built-in functions**
>
> Everything in `pyspark.sql.functions` runs in the JVM and is optimised. A Python UDF is slower and hidden from the optimizer.