Trumps Middle Name The Uncommon Story Of Cruz A Deep Dive Into The Details And Implications

Setting Up Cross-Environment Pipeline Secrets With SOPS and AWS KMS

Managing secrets across dev, staging, and production environments is one of those problems that seems simple until you've actually tried to solve it properly. I spent about six months working through this exact setup, mostly because I kept running into the same wall: my team wanted encrypted config files in git, but we also needed different encryption keys per environment, and we couldn't just use a single GPG key because the production pipeline runs in a sandboxed AWS account with no human interactive access. The tool that eventually worked was SOPS (Secrets OPerationS) from Mozilla, combined with AWS KMS for envelope encryption. The basic idea is straightforward: you store your secrets as YAML or JSON files in your repository, SOPS encrypts them with a key that you manage in KMS, and the encrypted files can be safely committed to version control. The decryption happens at pipeline time, not in CI config, which means your plaintext never lives in GitHub Actions logs or Jenkins console output. Here is how I actually got this working end-to-end, including the part that tripped me up for two weeks.

Installing SOPS and the necessary plugins

You need SOPS installed on your developer machines and on every CI runner that needs to decrypt files. On macOS: On Linux (Debian/Ubuntu): The tricky part is the KMS plugin. SOPS supports multiple backends, and the AWS KMS one requires you to have the AWS CLI configured with credentials that have permission to call kms:Decrypt and kms:Encrypt. If you are using AWS IAM Roles Anywhere or ECS task roles, make sure those roles include at minimum:

Do not attach policy to the key itself granting broad decrypt access. I learned that the hard way when someone pushed a misconfigured pipeline and nearly decrypted production secrets to a public artifact bucket. For each environment, create a separate KMS key. Do not reuse a single key across dev, staging, and production. Here is the command I used: The policy file is where most people get sloppy. Here is a working example that restricts usage to specific IAM roles and limits the key to a single region:

The region restriction is critical. Without it, if your credentials are compromised in a different region, the attacker can decrypt your secrets even if they cannot access your S3 bucket or ECR repository. This is the file that maps your repository paths to encryption keys. Place it at the root of your project. Here is a real example from a multi-environment setup: A few things to note here. The path_regex is evaluated against the file path relative to the .sops.yaml location. The empty age and pgp fields mean those backends are disabled for those paths. If you want to support multiple KMS keys for the same path (key rotation), you can list them comma-separated in the kms field.

I also learned that SOPS evaluates rules top-to-bottom and uses the first match. If you have a catch-all rule at the top, it will shadow everything below it. Put your most specific rules first.

Encrypting your first secret file

Start by creating a plaintext YAML file in the correct directory: Then encrypt it: The -i flag edits the file in place. If you omit it, SOPS prints the encrypted content to stdout instead. After encryption, the file will look something like this:

The file is now safe to commit. Anyone with the KMS key can decrypt it. Anyone without it sees only garbage. Here is where I hit the biggest problem. My production deployment runs inside an AWS CodePipeline that assumes an IAM role. That role has KMS decrypt permission, but there is no SSH agent, no GPG homedir, and no age key file. SOPS needs some form of identity to decrypt, and if you configure it wrong, the pipeline fails silently with a confusing error message. The solution is to let SOPS use the AWS SDK directly. In your GitHub Actions workflow, you do not need to set any SOPS-specific environment variables beyond the AWS credentials. The AWS SDK will automatically pick up the IAM role assumption from the ECS task or EC2 instance profile.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Decrypt secrets
        run: sops -d secrets/prod/database.yaml > /tmp/db-secret.yaml
        env:
          AWS_ROLE_ARN: arn:aws:iam::ACCOUNT:role/ci-deploy-role
          AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/kubernetes.io/serviceaccount/token

      - name: Deploy
        run: ./deploy.sh
        env:
          DB_PASSWORD: $(yq '.database.password' /tmp/db-secret.yaml)
          DB_HOST: $(yq '.database.host' /tmp/db-secret.yaml)

The web identity token approach works for EKS workloads. If you are using CodePipeline with CodeDeploy, you skip the role assumption step entirely because the pipeline execution role already has the necessary KMS permissions baked in. When you rotate a KMS key, existing encrypted files do not automatically update. The enc field inside the sops metadata block contains the old data key, encrypted with the old key material. If you rotate the KMS key in AWS but do not re-encrypt the files, SOPS will fail to decrypt them because it is looking for a data key that was encrypted under the previous version of the key. The workaround is to re-encrypt every file after key rotation. I wrote a script that iterates over all encrypted files in the repo and runs SOPS in-place update mode without changing the plaintext content:

find secrets -name "*.yaml" -exec sops -u -i {} \;-pre>

The -u flag tells SOPS to update the encryption keys in the metadata without modifying the encrypted data itself. This is significantly faster than re-encrypting from scratch because SOPS only touches the envelope, not the payload. After running that script, commit the updated files. Every developer and every CI runner will need the new KMS key version accessible, or they will get a kms:InvalidCiphertextException on the next pull.


Key rotation and access control best practices

Schedule automatic key rotation at the KMS level. In the AWS console, enable automatic rotation on each key, which creates a new key version every year. Your SOPS files will continue to decrypt because KMS retains the old key material and maps it to the same key ID. But automatic rotation only handles the encryption layer. You still need to manage who can call kms:Decrypt on each environment key. I recommend a strict model: Production KMS key: only the production deployment role and one security audit role can decrypt. No developer workstation has access.
Staging KMS key: the staging deployment role and any developer with the staging IAM role can decrypt. Dev KMS key: any developer with the default AWS profile can decrypt. This layered approach means that even if a developer credentials leak, the attacker only gets dev secrets, not production ones. It is a small thing, but I have seen teams treat all environments the same and then wonder why a leaked staging token gave someone access to production databases.
Common pitfalls

Accidentally committing plaintext files. SOPS creates a backup file with the extension .sops when you edit in place. Delete these immediately. They contain unencrypted content. Misconfigured path_regex. If your regex does not match the actual file path, SOPS falls back to the first matching rule or refuses to encrypt. Run sops --show-parse-errors to debug this. Using the same KMS key for multiple projects. It works technically, but you lose the ability to revoke access to one project without affecting all others. Create separate keys per project per environment.


Forgetting that SOPS uses AES-256-GCM for envelope encryption and a random data key per file. Each file gets its own data key. That is why key rotation does not break old files — the data key is stored encrypted under the current key version, and KMS can decrypt it with any version of the key.
Alternative tools to consider

If your stack does not run on AWS, SOPS supports GCP KMS, Azure Key Vault, HashiCorp Vault, and PGP. The configuration is nearly identical — just swap the backend in .sops.yaml. For non-cloud environments, I have used SOPS with age, which is a simpler key-based system that does not require a cloud account. It works well for small teams without AWS infrastructure. Another option is Sealed Secrets for Kubernetes, which encrypts secrets at the client side and only the cluster controller can decrypt them. It is more locked-in to Kubernetes but eliminates the need for KMS entirely during deployment. The bottom line is that SOPS with KMS gives you a good balance of portability, security, and operational simplicity. The setup takes about an hour for a single environment, and the ongoing maintenance is minimal. Just remember to rotate keys properly, keep your .sops.yaml clean, and never commit .sops backup files.