🚀 Walmart Data Engineering Project | End-to-End AWS & Snowflake Pipeline
In productionRecording in progress· 12 chapters
Recording in progress
The chapters and notes are ready; the video will appear here when it is published.
The promise
Watch an end-to-end retail batch pipeline take synthetic records from two databases, validate and transform them on AWS, load Snowflake, and refresh a dashboard — then learn how it was built across the series.
Episode 1 · target data flow
PostgreSQLorders · payments
MongoDBproducts · reviews
AWS GlueJDBC + connector
Amazon S3raw Parquet
AWS Lambdavalidate batch
PySparkjoins · KPIs
SnowflakeCOPY INTO + MERGE
Dashboardbatch refresh
Target architecture
Solid arrows are data flow; dashed arrows are orchestration — Step Functions coordinates jobs, it does not carry the data.
FlowchartSwipe to explore →Technical diagram
Rendering flowchart
What you will be able to answer
- 1
The business problem
What retail question do the KPIs answer, and why do orders and products live in different databases? - 2
Storage choices
Why land raw data in S3 as Parquet, in immutable batch paths? - 3
Trust
What do validation and DynamoDB checkpoints prevent when a batch fails or is replayed? - 4
Business logic
Where does PySpark add value — joins, de-duplication and retail KPIs? - 5
Loading without double counting
Why COPY INTO into a stage, then MERGE into analytics tables? - 6
Automation
Who schedules what: Airflow decides when, Step Functions coordinates AWS.
The series
| Episode | Topic |
|---|---|
| 1 | The finished pipeline and its architecture (this page) |
| 2 | Synthetic retail sources: Docker, Python, PostgreSQL and MongoDB |
| 3 | AWS Glue ETL from PostgreSQL and MongoDB to S3 Parquet |
| 4 | Lambda validation and DynamoDB checkpoints — recovering a failed batch |
| 5 | PySpark: from raw retail orders to sales KPIs |
| 6 | Snowflake COPY INTO + MERGE without double counting |
| 7 | Automating the pipeline with Airflow and Step Functions |
| 8 | Retail sales dashboard and the full pipeline test |
Honest scope
- Independently designed with synthetic data — not Walmart's architecture, and not sponsored by Walmart.
- Batch, automatically refreshed — not real time. Freshness will be shown as measured lag.
- Volumes, durations and "no duplicates" claims are published only after the replay and warehouse tests demonstrate them.