# Designing an S3 Data Lake Layout

> Raw, curated and published zones, partition keys, file sizes and naming conventions that keep a data lake fast, cheap and replayable.

- Source: https://akshaythakare.com/notes/aws/s3-data-lake-layout
- Topic: AWS
- Level: beginner
- Reading time: 2 min
- Tags: aws, s3, data-lake, partitioning, interview
- Updated: 2026-10-11

## Zones

```text
s3://company-lake/
  raw/         source data as received (JSONL / Parquet), immutable
  curated/     cleaned, typed, deduplicated Parquet
  published/   aggregates ready for the warehouse or BI
```

| Zone | Who writes | Rule |
| --- | --- | --- |
| raw | Ingestion jobs | Never overwrite; one prefix per batch or run |
| curated | Transform jobs | Rewritten per partition; schema enforced |
| published | Aggregation jobs | Shaped for consumers |

## Partition keys

```text
curated/sales/business_date=2026-01-05/part-0000.parquet
```

- Partition by what queries filter on, usually a date.
- Avoid high-cardinality keys such as `customer_id`, which create millions of tiny folders.
- Use `key=value` folder names so Glue, Athena and Spark discover partitions automatically.

## File sizes

Aim for roughly 128 MB–1 GB per file. Thousands of small files make every query slow, because opening a file costs more than reading it.

## Make raw replayable

```text
raw/orders/run_id=7c1e…/part-0000.jsonl
raw/orders/run_id=7c1e…/manifest.json     # files, row counts, checksums
```

A manifest lists exactly which files belong to a run, so a replay never picks up partial or stray files.

> **Note: Security basics**
>
> Block public access, encrypt by default, enable versioning on raw, and set lifecycle rules to expire or archive old raw data.