IAM for Data Pipelines: Least Privilege in Practice

Roles, not keys

Pipelines should never use long-lived access keys. Each service assumes a role:

A least-privilege policy

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::lake/raw/*" },
    { "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": "arn:aws:s3:::lake/curated/*" },
    { "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": "arn:aws:s3:::lake",
      "Condition": { "StringLike": { "s3:prefix": ["raw/*", "curated/*"] } } }
  ]
}

• Scope actions (GetObject, not s3:*).
• Scope resources to prefixes, not the whole bucket.
• ListBucket applies to the bucket ARN, limited by a prefix condition.

Secrets

Database passwords and API keys belong in Secrets Manager (or SSM Parameter Store), read at runtime by the role. They should never be in code, job parameters or environment files committed to git.

How to debug AccessDenied

1. Which role made the call? Check CloudTrail.
2. Is the action and the exact resource ARN allowed?
3. Is there an explicit Deny (bucket policy, SCP, KMS key policy)?
4. For encrypted buckets, does the role also have kms:Decrypt?