# Python UDFs vs Built-in Functions vs Pandas UDFs

> Why Python UDFs are slow, when a pandas UDF is acceptable, and how to rewrite most UDFs with built-in functions.

- Source: https://akshaythakare.com/notes/pyspark/udfs-vs-built-in-functions
- Topic: PySpark
- Level: intermediate
- Reading time: 2 min
- Tags: pyspark, performance, udf, interview
- Updated: 2026-10-11

## What happens in a Python UDF

Spark runs in the JVM. For a regular Python UDF, each row is serialised, sent to a Python worker, processed and sent back. The optimizer cannot see inside the function, so it cannot push filters through it or prune columns.

| Option | Speed | Use when |
| --- | --- | --- |
| Built-in (`F.when`, `F.regexp_extract`, …) | Fastest | Almost always possible |
| Pandas UDF (vectorised, Arrow) | Moderate | Logic needs a Python library (e.g. scoring a model) |
| Python UDF (row at a time) | Slowest | Last resort |

## Rewrite with built-ins

```python
# UDF
@F.udf("string")
def band(amount):
    return "high" if amount > 1000 else "low"

# built-in: same result, optimised
df = df.withColumn("band", F.when(F.col("amount") > 1000, "high").otherwise("low"))
```

Common replacements: `F.regexp_extract`, `F.split`, `F.coalesce`, `F.date_trunc`, `F.transform` (for arrays) and `F.from_json`.

## When you need Python: pandas UDF

```python
import pandas as pd

@F.pandas_udf("double")
def score(amount: pd.Series, qty: pd.Series) -> pd.Series:
    return model.predict(pd.DataFrame({"amount": amount, "qty": qty}))
```

It processes whole Arrow batches, so serialisation cost is shared across thousands of rows.

> **Warning: Nulls and types**
>
> A UDF receives Python `None` for nulls. Handle it, and declare the return type correctly. A wrong return type silently produces nulls.