Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

ETL / ELT

Ingestion patterns, idempotency, replay and data quality.

5 notes · all topicsData engineering

Start here · beginner1

  1. 1ETL vs ELT: Where the Transformation HappensThe difference between transforming before or after loading, why cloud warehouses pushed teams to ELT, and when ETL still wins.

Next · intermediate3

  1. 2Incremental Loads and WatermarksLoad only what changed since the last run: high-water marks, choosing the right column, overlap windows and the rows a naive watermark misses.
  2. 3Designing Idempotent, Replayable PipelinesBatch IDs, immutable attempts, manifests, ledgers and watermarks — the building blocks that make 'just re-run it' safe.
  3. 4Change Data Capture: Moving Only What ChangedQuery-based and log-based CDC compared — what each catches, what each misses, and how to apply changes downstream without duplicates.

Go deeper · advanced1

  1. 5Late-Arriving Data: Facts and DimensionsWhat to do when a fact arrives after its day was processed, or before its dimension exists: reprocessing windows, placeholders and reconciliation.