Skip to content
PHP2Shell: The PHP Bug
Akshay Thakare

Data Engineering System Design

Designing pipelines and platforms end to end, with trade-offs.

4 notes · all topicsCloud & orchestration

Next · intermediate3

  1. 1Data Pipeline Observability: Freshness, Volume and QualityWhat to monitor so you find broken data before your users do: freshness, volume, schema, quality checks, lineage and alert design.
  2. 2System Design: A Daily Batch Pipeline, Step by StepA repeatable answer for the most common data engineering design question: requirements, architecture, data model, failure handling and monitoring.
  3. 3Batch vs Streaming: Choosing the Right PipelineLatency requirements, correctness, cost and operations — a practical framework for the most common data system design question.

Go deeper · advanced1

  1. 4Exactly-Once Is a Myth; Idempotency Is the AnswerWhy distributed systems deliver at-least-once, what 'exactly-once' really means in Kafka and Spark, and how idempotent writes give you correct results.