Skip to main content

Amazon S3 destination

Subsets delivers the export feeds into an S3 bucket you own. Setup happens in the app under Settings → Data feeds: it walks you through the bucket, the IAM policy and the IAM role, generates both policy documents, and validates the connection. This page covers what to have ready and what your cloud or security team will ask.

Have ready​

  • The 12-digit AWS account ID of the account holding the bucket. Subsets fills it into the trust policy; you never paste an ARN.
  • An S3 bucket, any region - it is detected automatically, with a Region field as fallback. Optionally a key prefix such as subsets/ to group the files; blank delivers to the bucket root. One destination per organization.
  • Someone who can create an IAM policy and an IAM role in that account. The app generates both documents; the names are fixed - subsets-data-export-policy and subsets-data-export-role - so check your naming rules allow them.
  • Subsets organization admin. Non-admins see Data feeds read-only.
  • About 15 minutes. Setup ends with Subsets writing <prefix>_validation/subsets-validation.txt to your bucket; you open it with your own credentials and paste the code back. Feeds switch on only after this passes, and later changes to account, bucket, prefix or region pause deliveries until you re-validate.

For your cloud or security team​

Use the generated policies​

Do not hand-write them. The app fills in your account ID, bucket and prefix, the Subsets service-account ID and your organization's session name; the last two are published nowhere else, so a hand-written policy will not validate. Both documents stay under Destination after setup.

To author the role in Terraform or CloudFormation, paste the two documents into your template verbatim. Common mistakes:

  • The trust condition key is accounts.google.com:oaud, not aud. For Google tokens AWS maps aud to the token's azp claim; oaud is the aud claim Subsets sets, and it must equal the role's own ARN.
  • No IAM OIDC identity-provider resource is needed - accounts.google.com is built in.
  • The prefix in Resource must match the form the app stores: no leading slash, one trailing slash, or empty for the bucket root.

How access works​

  • Keyless. Per delivery, the Subsets service account exchanges a Google-signed OIDC token at AWS STS (AssumeRoleWithWebIdentity) for one-hour credentials on a single role in your account. There are no access keys to store or rotate.
  • Write-only. The role grants s3:PutObject under your prefix and nothing else. It cannot list or read, so you cannot test it by reading back what Subsets wrote - the in-app validation step is the connectivity check.
  • Pinned three ways. The trust policy requires the exact Subsets service account (accounts.google.com:sub), your organization as the session name (sts:RoleSessionName), and an audience (oaud) equal to your own role's ARN - so a token minted for any other role cannot be replayed against yours.
  • Revocable by you. Delete the role and deliveries stop.
  • A bucket policy cannot replace the role: bucket policies cannot grant access to a Google identity.

Encryption​

Default SSE-S3 (AES256) needs nothing extra. If the bucket enforces SSE-KMS with a customer-managed key, the role also needs kms:GenerateDataKey and kms:Decrypt on that key, plus a matching statement in the key policy. Validation reports when this is why a write was refused.