Writing

Synchronizing Kubernetes Secrets with AWS Secrets Manager using Secrets Store CSI Driver

In early-stage teams, sensitive environment configuration—API tokens, database credentials, encryption salts—frequently ends up stored in uncommitted secret.yaml files passed between developers via direct messaging.

This approach creates severe risks:

  1. Plaintext vulnerability: Kubernetes Secrets are merely Base64-encoded strings, not encrypted at rest by default.
  2. Synchronization drift: When team members update credentials manually, running clusters diverge, leading to deployment outages.

To solve this, we migrated our cluster secret lifecycle to AWS Secrets Manager synchronized natively into pods using the Secrets Store CSI Driver.

Architecture Overview

Instead of bundling proprietary AWS SDKs into every application container (tightly coupling application code to cloud providers), the Kubernetes CSI Driver intercepts volume mounts at pod startup:

  1. A pod mounts a virtual CSI volume configured via a SecretProviderClass.
  2. The AWS Secrets Store CSI provider authenticates using IAM Roles for Service Accounts (IRSA).
  3. The provider fetches the secret from AWS Secrets Manager and writes it to the pod's volume mount.
  4. Optionally, the driver mirrors the secret into a native Kubernetes Secret object, allowing you to inject values as standard environment variables via envFrom.secretRef.

Implementation Guide

1. Install Secrets Store CSI Driver and AWS Provider

Deploy the driver using Helm with secret synchronization and rotation flags enabled:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws

# Install driver with secret synchronization enabled
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true \
  --set enableSecretRotation=true

# Install AWS provider daemonset
helm install secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws \
  --namespace kube-system

2. Configure AWS IAM and ServiceAccount (IRSA)

Create an IAM policy granting read access to specific secrets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:ap-northeast-2:123456789012:secret:production/app/*"
    }
  ]
}

Bind the IAM Role to a Kubernetes ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-service-account
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/k8s-secrets-reader-role

3. Define the SecretProviderClass

The SecretProviderClass defines which remote AWS secret object to pull and how to map internal JSON keys into a synchronized Kubernetes Secret:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-aws-secrets
  namespace: default
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "production/app/credentials"
        objectType: "secretsmanager"
  secretObjects:
    - secretName: app-synced-secret
      type: Opaque
      data:
        - objectName: "production/app/credentials"
          key: USERNAME
        - objectName: "production/app/credentials"
          key: PASSWORD

4. Mounting into the Deployment

The volume mount is mandatory because the CSI driver triggers secret fetching during volume mounting:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      serviceAccountName: api-service-account
      containers:
        - name: app
          image: mycompany/api:v1.0.0
          # Injected cleanly as standard environment variables!
          envFrom:
            - secretRef:
                name: app-synced-secret
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "app-aws-secrets"

Key Benefits

  • Zero application coupling: The app reads standard POSIX environment variables without importing AWS client SDKs or managing credentials in runtime code.
  • Centralized auditing & rotation: AWS CloudTrail logs every secret access, and Secrets Manager's rotation lambda updates credentials on schedule without pod redeployments.
  • Clean GitOps: Deployment manifests commit safely to public or private git repositories because no sensitive values reside in code.