AWS
Glue, EMR, S3, Lambda, Step Functions and the data services around them.
Start here · beginner2
- 1Designing an S3 Data Lake LayoutRaw, curated and published zones, partition keys, file sizes and naming conventions that keep a data lake fast, cheap and replayable.2 min
- 2Glue Data Catalog, Crawlers and AthenaHow the Glue Data Catalog turns S3 folders into queryable tables, when to use crawlers vs explicit DDL, and querying with Athena.2 min
Next · intermediate3
- 3IAM for Data Pipelines: Least Privilege in PracticeRoles vs users, how a Glue job or Lambda gets permissions, writing a least-privilege policy, and keeping secrets out of code.2 min
- 4AWS Glue JDBC Ingestion: What You Must Get RightConnections, VPC networking, secrets, parallel reads and bookmarks — the checklist for pulling relational data into S3 with Glue.3 min
- 5Step Functions vs Airflow — and Using BothAirflow is a scheduler and dependency manager; Step Functions is a durable AWS-native state machine. Here is how to split responsibilities.3 min