Delta Lake: ACID Tables on a Data Lake

Plain Parquet has no transactions

Two jobs writing the same folder, a job that dies halfway, a reader that lists files mid-write — on a raw data lake every one of these can produce partial or duplicated data.

The transaction log

A Delta table is Parquet files plus a _delta_log/ folder of ordered JSON commits. Readers only see files that a committed log entry adds.

sales/
├── part-0001.parquet
├── part-0002.parquet
└── _delta_log/
    ├── 00000000000000000000.json   # add part-0001
    ├── 00000000000000000001.json   # add part-0002
    └── 00000000000000000002.json   # remove part-0001, add part-0003  (MERGE)

1. 1

   Write new files

   The writer creates new Parquet files. Nobody can see them yet.
2. 2

   Commit atomically

   It tries to create the next log file (e.g. …003.json). Only one writer can succeed — that is the commit.
3. 3

   Readers use a snapshot

   A reader picks the latest committed version and reads exactly the files it lists.
4. 4

   Conflicts are detected

   If another writer committed first, Delta checks whether the two changes overlap and retries or fails.

Features you get from the log

from delta.tables import DeltaTable
 
target = DeltaTable.forName(spark, "silver.customers")
(target.alias("t")
   .merge(updates.alias("s"), "t.customer_id = s.customer_id")
   .whenMatchedUpdateAll()
   .whenNotMatchedInsertAll()
   .execute())