# AWS Glue JDBC Ingestion: What You Must Get Right

> Connections, VPC networking, secrets, parallel reads and bookmarks — the checklist for pulling relational data into S3 with Glue.

- Source: https://akshaythakare.com/notes/aws/glue-jdbc-ingestion
- Topic: AWS
- Level: intermediate
- Reading time: 3 min
- Tags: aws, glue, ingestion, jdbc
- Updated: 2026-10-11

## The moving parts

```mermaid
flowchart LR
  G[Glue job<br/>Spark workers] -->|ENI in private subnet| DB[(RDS / PostgreSQL)]
  G -->|VPC endpoint or NAT| S3[(S3)]
  G --> SM[Secrets Manager]
  G --> CW[CloudWatch Logs]
```

A Glue **connection** of type JDBC tells Glue which VPC, subnet and security group to place its workers' network interfaces in. Get networking wrong and the job hangs until timeout.

## Networking checklist

- The subnet has a route to the database and to **S3** (gateway VPC endpoint is cheapest) and to any other AWS APIs the job calls (Secrets Manager, CloudWatch — via interface endpoints or NAT)
- The security group has a **self-referencing inbound rule** for all TCP (Glue workers talk to each other)
- The database security group allows the Glue security group on the DB port
- Require TLS to the database where supported

## Credentials

Store credentials in **Secrets Manager** and reference them from the connection. Never pass passwords through job arguments or Step Functions input — those end up in logs and execution history.

## Reading in parallel

A plain JDBC read uses **one** connection and one partition. For large tables, split the read:

```python
df = (spark.read.format("jdbc")
        .option("url", jdbc_url)
        .option("dbtable", "public.orders")
        .option("partitionColumn", "order_seq")   # numeric/date column with even spread
        .option("lowerBound", 1).option("upperBound", 50_000_000)
        .option("numPartitions", 16)
        .option("fetchsize", 10_000)
        .load())
```

`lowerBound`/`upperBound` only decide partition **stride** — they do not filter rows. Keep `numPartitions` within what the source database can handle.

## Full snapshot vs incremental

- **Full snapshot** — simplest and naturally includes unchanged lookups; fine for small tables
- **Incremental by `updated_at`** — read `updated_at >= W - overlap AND updated_at < U`, and only advance the watermark after the downstream publish commits. Requires every writer to maintain `updated_at` and cannot see hard deletes
- **Glue job bookmarks** track progress for you, but you still need to reason about late commits and replay

> **Warning: Two sources are not one snapshot**
>
> Reading PostgreSQL and then MongoDB in one job does not give you a consistent cross-database snapshot. Document the skew and reconcile.