# 🚀 Walmart Data Engineering Project | End-to-End AWS & Snowflake Pipeline

> Episode 1 of the Walmart data engineering series: synthetic PostgreSQL and MongoDB data flows through AWS Glue, Amazon S3, Lambda, DynamoDB, PySpark, Snowflake, Step Functions and Airflow into a sales dashboard. An independent educational project — not Walmart's proprietary infrastructure. In production.

- Source: https://akshaythakare.com/videos/walmart-retail-pipeline-episode-01
- Technologies: PostgreSQL, MongoDB, Docker, AWS Glue, Amazon S3, AWS Lambda, DynamoDB, PySpark, Snowflake, Step Functions, Apache Airflow, Amazon SNS
- Updated: 2026-10-25

Episode 1 of the Walmart data engineering series: synthetic PostgreSQL and MongoDB data flows through AWS Glue, Amazon S3, Lambda, DynamoDB, PySpark, Snowflake, Step Functions and Airflow into a sales dashboard. An independent educational project — not Walmart's proprietary infrastructure. In production.

## Chapters

- 0:00 The Finished Pipeline in Action
- 1:00 The Walmart Retail Problem
- 2:15 Complete Architecture Explained
- 4:00 EC2, Docker, PostgreSQL & MongoDB
- 6:00 AWS Glue Ingestion to S3 Parquet
- 8:15 Lambda Validation & DynamoDB
- 9:30 PySpark Transformations & Retail KPIs
- 11:45 Snowflake COPY INTO & MERGE
- 13:45 Airflow, Step Functions & SNS
- 15:15 Sales Dashboard & End-to-End Test
- 16:45 Limitations, Costs & Engineering Lessons
- 17:30 Next Episode: Build the Source Systems

> **Warning: In production**
>
> This episode is planned and scripted; the cloud pipeline it demonstrates is being built now. Chapter times are the planned 18-minute edit. Results, timings and links will appear here only after they have been captured from the real run.

## The promise

Watch an end-to-end retail **batch** pipeline take synthetic records from two databases, validate and transform them on AWS, load Snowflake, and refresh a dashboard — then learn how it was built across the series.

**Episode 1 · target data flow:** PostgreSQL (orders · payments) → MongoDB (products · reviews) → AWS Glue (JDBC + connector) → Amazon S3 (raw Parquet) → AWS Lambda (validate batch) → PySpark (joins · KPIs) → Snowflake (COPY INTO + MERGE) → Dashboard (batch refresh)

## Target architecture

Solid arrows are data flow; dashed arrows are orchestration — Step Functions coordinates jobs, it does not carry the data.

```mermaid
flowchart LR
  subgraph SRC[Docker on EC2 · demo sources]
    PG[(PostgreSQL)]
    MG[(MongoDB)]
    GEN[Python generator] --> PG
    GEN --> MG
  end
  PG --> G1[Glue ingestion]
  MG --> G1
  G1 --> RAW[(S3 raw Parquet)]
  RAW --> VAL[Lambda validation]
  VAL <--> DDB[(DynamoDB checkpoints)]
  RAW --> G2[Glue PySpark KPIs]
  G2 --> CUR[(S3 curated)]
  CUR --> LD[Snowflake COPY INTO]
  LD --> MRG[MERGE analytics tables]
  MRG --> BI[Sales dashboard]
  AF[Airflow] -.-> SF[Step Functions]
  SF -.-> G1
  SF -.-> VAL
  SF -.-> G2
  SF -.-> LD
  SF -.-> SNS[SNS alerts]
```

## What you will be able to answer

  
### The business problem
What retail question do the KPIs answer, and why do orders and products live in different databases?
  
### Storage choices
Why land raw data in S3 as Parquet, in immutable batch paths?
  
### Trust
What do validation and DynamoDB checkpoints prevent when a batch fails or is replayed?
  
### Business logic
Where does PySpark add value — joins, de-duplication and retail KPIs?
  
### Loading without double counting
Why COPY INTO into a stage, then MERGE into analytics tables?
  
### Automation
Who schedules what: Airflow decides when, Step Functions coordinates AWS.

## The series

| Episode | Topic |
| --- | --- |
| 1 | The finished pipeline and its architecture (this page) |
| 2 | Synthetic retail sources: Docker, Python, PostgreSQL and MongoDB |
| 3 | AWS Glue ETL from PostgreSQL and MongoDB to S3 Parquet |
| 4 | Lambda validation and DynamoDB checkpoints — recovering a failed batch |
| 5 | PySpark: from raw retail orders to sales KPIs |
| 6 | Snowflake COPY INTO + MERGE without double counting |
| 7 | Automating the pipeline with Airflow and Step Functions |
| 8 | Retail sales dashboard and the full pipeline test |

## Honest scope

- Independently designed with **synthetic data** — not Walmart's architecture, and not sponsored by Walmart.
- **Batch**, automatically refreshed — not real time. Freshness will be shown as measured lag.
- Volumes, durations and "no duplicates" claims are published only after the replay and warehouse tests demonstrate them.