Glue Data Catalog, Crawlers and Athena
How the Glue Data Catalog turns S3 folders into queryable tables, when to use crawlers vs explicit DDL, and querying with Athena.
AWSBeginner2 min readUpdated 11 Oct 2026
AWSThe catalog is metadata
S3 stores files. The Glue Data Catalog stores table definitions: column names and types, file format, S3 location and partitions. Athena, Glue jobs, EMR and Redshift Spectrum all read the same catalog.
Crawler or DDL?
| Approach | Good for | Risk |
|---|---|---|
| Crawler | Exploring new data quickly | Guesses types; can create odd tables from stray files |
| Explicit DDL | Production tables | You maintain the schema deliberately |
CREATE EXTERNAL TABLE curated.sales (
store_id int, dept_id int, net_sales decimal(18,2)
)
PARTITIONED BY (business_date date)
STORED AS PARQUET
LOCATION 's3://lake/curated/sales/';Partitions must be registered
New business_date=… folders are not visible until the catalog knows them:
MSCK REPAIR TABLE curated.sales; -- scans for new partitions (slow on many)
ALTER TABLE curated.sales ADD PARTITION (business_date='2026-01-05')
LOCATION 's3://lake/curated/sales/business_date=2026-01-05/';Partition projection avoids registration entirely for predictable keys such as dates.
Querying with Athena
SELECT business_date, SUM(net_sales)
FROM curated.sales
WHERE business_date BETWEEN DATE '2026-01-01' AND DATE '2026-01-31'
GROUP BY 1;Athena charges per byte scanned. Parquet plus a partition filter can cut cost by over 90%.
Finished this note?