AWS Glue JDBC Ingestion: What You Must Get Right

The moving parts

A Glue connection of type JDBC tells Glue which VPC, subnet and security group to place its workers' network interfaces in. Get networking wrong and the job hangs until timeout.

Networking checklist

• The subnet has a route to the database and to S3 (gateway VPC endpoint is cheapest) and to any other AWS APIs the job calls (Secrets Manager, CloudWatch — via interface endpoints or NAT)
• The security group has a self-referencing inbound rule for all TCP (Glue workers talk to each other)
• The database security group allows the Glue security group on the DB port
• Require TLS to the database where supported

Credentials

Store credentials in Secrets Manager and reference them from the connection. Never pass passwords through job arguments or Step Functions input — those end up in logs and execution history.

Reading in parallel

A plain JDBC read uses one connection and one partition. For large tables, split the read:

df = (spark.read.format("jdbc")
        .option("url", jdbc_url)
        .option("dbtable", "public.orders")
        .option("partitionColumn", "order_seq")   # numeric/date column with even spread
        .option("lowerBound", 1).option("upperBound", 50_000_000)
        .option("numPartitions", 16)
        .option("fetchsize", 10_000)
        .load())

lowerBound/upperBound only decide partition stride — they do not filter rows. Keep numPartitions within what the source database can handle.

Full snapshot vs incremental

• Full snapshot — simplest and naturally includes unchanged lookups; fine for small tables
• Incremental by updated_at — read updated_at >= W - overlap AND updated_at < U, and only advance the watermark after the downstream publish commits. Requires every writer to maintain updated_at and cannot see hard deletes
• Glue job bookmarks track progress for you, but you still need to reason about late commits and replay