Designing an S3 Data Lake Layout
Raw, curated and published zones, partition keys, file sizes and naming conventions that keep a data lake fast, cheap and replayable.
AWSBeginner2 min readUpdated 11 Oct 2026
AWSZones
s3://company-lake/
raw/ source data as received (JSONL / Parquet), immutable
curated/ cleaned, typed, deduplicated Parquet
published/ aggregates ready for the warehouse or BI| Zone | Who writes | Rule |
|---|---|---|
| raw | Ingestion jobs | Never overwrite; one prefix per batch or run |
| curated | Transform jobs | Rewritten per partition; schema enforced |
| published | Aggregation jobs | Shaped for consumers |
Partition keys
curated/sales/business_date=2026-01-05/part-0000.parquet- Partition by what queries filter on, usually a date.
- Avoid high-cardinality keys such as
customer_id, which create millions of tiny folders. - Use
key=valuefolder names so Glue, Athena and Spark discover partitions automatically.
File sizes
Aim for roughly 128 MB–1 GB per file. Thousands of small files make every query slow, because opening a file costs more than reading it.
Make raw replayable
raw/orders/run_id=7c1e…/part-0000.jsonl
raw/orders/run_id=7c1e…/manifest.json # files, row counts, checksumsA manifest lists exactly which files belong to a run, so a replay never picks up partial or stray files.
Finished this note?