How to Store Secrets Securely in Kubernetes

The Default Kubernetes Secret Model Is Not a Final Answer

Kubernetes Secret objects are useful, but they are not a complete secrets strategy. They are base64-encoded objects stored in etcd, and they inherit all the strengths and weaknesses of the surrounding cluster design.

That means the real question is not "Should we use Kubernetes secrets?" It is "What level of exposure are we accepting if the cluster, backup path, or access model fails?"

When Native Secrets Are Acceptable

Native secrets can be acceptable for lower-risk cases when all of the following are true:

• etcd encryption at rest is enabled
• RBAC is tight
• secret access is scoped to specific workloads
• backups are protected
• rotation is manageable

Even then, they are rarely the best long-term answer for high-value credentials.

Better Patterns for Production

Pattern 1: External Secrets Operator

This is often the most practical production model.

The application still reads a Kubernetes secret, but the source of truth lives in Vault, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager.

Benefits:

• central secret lifecycle
• rotation outside the manifest repo
• better separation between cluster config and credential authority

Pattern 2: Vault for Dynamic or Short-Lived Secrets

If you need database credentials, PKI, or tightly controlled secret issuance, Vault becomes more compelling.

Pattern 3: Sealed Secrets for GitOps Workflows

Useful when you need encrypted values in Git, but remember the cluster key model carefully. It protects secrets in transit through Git, not every downstream exposure path.

What Not to Do

Still common and still risky:

• storing plaintext credentials in Helm values
• keeping secrets in base64 inside repo manifests and calling it secure
• granting broad namespace read access to operators or support tooling
• reusing one database password across many services for convenience

Example: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: app-db
  data:
    - secretKey: password
      remoteRef:
        key: prod/database/app
        property: password

This is better than manually copying secrets into YAML because the source of truth stays in the manager built for that job.

Rotation Matters More Than Storage Format

Ask these questions:

• how fast can this secret rotate after a suspected leak?
• who owns that rotation process?
• how many workloads break if it changes?

Case pattern:

Teams often get stuck because one credential is shared by too many services. The storage mechanism is not the real problem. The dependency design is.

RBAC and Namespace Design Still Decide the Risk

Even with external secret integration, you still need:

• namespace boundaries that match trust boundaries
• service accounts scoped narrowly
• no blanket get secrets access for convenience
• audit logs around secret reads and secret-management changes where possible

Incident Response for Kubernetes Secrets

If a cluster or namespace is suspected to be compromised:

1. identify which secrets were reachable
2. rotate the highest-risk credentials first
3. invalidate tokens, certificates, and cloud keys where relevant
4. confirm dependent services recovered cleanly
5. review why that secret was reachable from the affected workload at all

Production Checklist

• etcd encryption at rest enabled
• namespace and service account access minimized
• high-value secrets sourced from external manager
• rotation path documented and tested
• GitOps path encrypts secret material properly
• shared secrets reduced where possible
• incident playbook exists for secret exposure

Further Reading

• Kubernetes Secrets Documentation
• External Secrets Operator
• Bitnami Sealed Secrets
• HashiCorp Vault Kubernetes Auth

Related SecureCodeReviews guides:

• Kubernetes Security Best Practices
• Secrets Management in DevSecOps
• Cloud Security Assessment Checklist

The safest way to store secrets in Kubernetes is usually not to treat the cluster as the primary secret authority at all. Let the cluster consume secrets, not own their full lifecycle.