# End-to-End Data Engineering Project: PostgreSQL & MongoDB → AWS Glue → S3 → Snowflake

> A chaptered walkthrough of a production-style batch pipeline: Glue JDBC and MongoDB ingestion, Lambda manifest validation, PySpark cleaning and joins, Snowflake COPY INTO, customer history with safe replay, and Airflow + Step Functions orchestration.

- Source: https://akshaythakare.com/videos/aws-snowflake-pipeline-walkthrough
- Technologies: AWS Glue, PySpark, Amazon S3, AWS Lambda, Snowflake, Apache Airflow, Step Functions, PostgreSQL, MongoDB, Power BI
- Updated: 2026-10-11

A chaptered walkthrough of a production-style batch pipeline: Glue JDBC and MongoDB ingestion, Lambda manifest validation, PySpark cleaning and joins, Snowflake COPY INTO, customer history with safe replay, and Airflow + Step Functions orchestration.

## Chapters

- 0:00 The business problem and architecture
- 2:00 Environment, sources and S3
- 5:30 Glue JDBC and MongoDB ingestion
- 9:30 Lambda manifest validation
- 12:30 PySpark cleaning, joins and quality
- 17:30 S3 to Snowflake with COPY INTO
- 21:00 Facts, customer history and safe replay
- 24:30 Airflow and Step Functions orchestration
- 28:00 Power BI, cleanup and takeaways

## What this video covers

How relational orders and document customer profiles become **trusted business metrics** — and how retries avoid duplicate sales. The video demonstrates one successful batch, a changed customer, a replay that does not duplicate facts, and a failed validation with recovery.

> **Note: Recording status**
>
> The script, chapters and narration are prepared; the YouTube upload is not yet published. Chapter times are planned estimates and will be updated to the measured timings at publish.

## Follow along

1. Clone the repository and run `python3 -m pipeline.local demo`
2. Read the [full case study](/projects/aws-snowflake-pipeline) beside the video
3. Practice the interview questions at the end of the case study