# Glue Data Catalog, Crawlers and Athena

> How the Glue Data Catalog turns S3 folders into queryable tables, when to use crawlers vs explicit DDL, and querying with Athena.

- Source: https://akshaythakare.com/notes/aws/glue-data-catalog
- Topic: AWS
- Level: beginner
- Reading time: 2 min
- Tags: aws, glue, athena, data-catalog
- Updated: 2026-10-11

## The catalog is metadata

S3 stores files. The **Glue Data Catalog** stores table definitions: column names and types, file format, S3 location and partitions. Athena, Glue jobs, EMR and Redshift Spectrum all read the same catalog.

## Crawler or DDL?

| Approach | Good for | Risk |
| --- | --- | --- |
| Crawler | Exploring new data quickly | Guesses types; can create odd tables from stray files |
| Explicit DDL | Production tables | You maintain the schema deliberately |

```sql
CREATE EXTERNAL TABLE curated.sales (
  store_id int, dept_id int, net_sales decimal(18,2)
)
PARTITIONED BY (business_date date)
STORED AS PARQUET
LOCATION 's3://lake/curated/sales/';
```

## Partitions must be registered

New `business_date=…` folders are not visible until the catalog knows them:

```sql
MSCK REPAIR TABLE curated.sales;          -- scans for new partitions (slow on many)
ALTER TABLE curated.sales ADD PARTITION (business_date='2026-01-05')
  LOCATION 's3://lake/curated/sales/business_date=2026-01-05/';
```

Partition projection avoids registration entirely for predictable keys such as dates.

## Querying with Athena

```sql
SELECT business_date, SUM(net_sales)
FROM curated.sales
WHERE business_date BETWEEN DATE '2026-01-01' AND DATE '2026-01-31'
GROUP BY 1;
```

Athena charges per byte scanned. Parquet plus a partition filter can cut cost by over 90%.