# Delta Lake: ACID Tables on a Data Lake

> How the transaction log turns Parquet files into tables with atomic writes, time travel, MERGE and schema enforcement.

- Source: https://akshaythakare.com/notes/databricks/delta-lake-acid-tables
- Topic: Databricks
- Level: intermediate
- Reading time: 2 min
- Tags: databricks, delta-lake, lakehouse, acid
- Updated: 2026-10-11

## Plain Parquet has no transactions

Two jobs writing the same folder, a job that dies halfway, a reader that lists files mid-write — on a raw data lake every one of these can produce partial or duplicated data.

## The transaction log

A Delta table is Parquet files **plus** a `_delta_log/` folder of ordered JSON commits. Readers only see files that a committed log entry adds.

```text
sales/
├── part-0001.parquet
├── part-0002.parquet
└── _delta_log/
    ├── 00000000000000000000.json   # add part-0001
    ├── 00000000000000000001.json   # add part-0002
    └── 00000000000000000002.json   # remove part-0001, add part-0003  (MERGE)
```

  
### Write new files
The writer creates new Parquet files. Nobody can see them yet.
  
### Commit atomically
It tries to create the next log file (e.g. `…003.json`). Only one writer can succeed — that is the commit.
  
### Readers use a snapshot
A reader picks the latest committed version and reads exactly the files it lists.
  
### Conflicts are detected
If another writer committed first, Delta checks whether the two changes overlap and retries or fails.

## Features you get from the log

| Feature | Example |
| --- | --- |
| Atomic overwrite / append | A failed job leaves the table unchanged |
| Time travel | `SELECT * FROM sales VERSION AS OF 12` |
| MERGE (upsert) | Apply CDC or SCD logic in one statement |
| Schema enforcement | Writes with unexpected columns fail |
| Schema evolution | `.option("mergeSchema", "true")` when you mean it |

```python
from delta.tables import DeltaTable

target = DeltaTable.forName(spark, "silver.customers")
(target.alias("t")
   .merge(updates.alias("s"), "t.customer_id = s.customer_id")
   .whenMatchedUpdateAll()
   .whenNotMatchedInsertAll()
   .execute())
```

> **Warning: Housekeeping matters**
>
> Run `OPTIMIZE` to compact small files and `VACUUM` to delete files no longer referenced. Vacuuming too aggressively removes the history that time travel depends on.