📘 Vault + External Secrets Operator (ESO) + ENV Injection Demo

This documentation provides a step-by-step integration of HashiCorp Vault with External Secrets Operator (ESO) to securely inject secrets as environment variables into Kubernetes Pods. This setup ensures security best practices for managing application secrets at runtime.

📂 Project Structure

vault-eso-demo/
├── README.md
├── k8s/
│   ├── 00-namespace.yaml
│   ├── 01-serviceaccount.yaml
│   ├── 02-secretstore.yaml
│   ├── 03-externalsecret.yaml
│   ├── 04-nginx-deployment.yaml
└── vault/
    ├── policy.hcl
    └── vault-setup.sh

🔧 Prerequisites

Component	Required
Kubernetes Cluster	✅
Helm CLI	✅
HashiCorp Vault (dev mode ok)	✅
External Secrets Operator	✅

🚀 Setup Walkthrough

1. 📦 Install External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
 
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --set webhook.certManager.enabled=false \
  --set webhook.selfSigned.enabled=true

Validate installation:

kubectl get pods -n external-secrets -o wide
kubectl get crds | grep external-secrets.io
kubectl api-resources | grep -i external

2. 🔐 Vault Configuration (dev mode)

All commands assume execution inside the Vault pod or with the Vault CLI set up locally.

vault auth enable kubernetes
 
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://${KUBERNETES_PORT_443_TCP_ADDR}:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Create policy: vault/policy.hcl

path "secret/data/myapp/env" {
  capabilities = ["read"]
}

Apply policy and create role:

vault policy write myapp-policy vault/policy.hcl
 
vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=external-secrets-sa \
  bound_service_account_namespaces=default \
  policies=myapp-policy \
  ttl=24h

Create test secrets:

vault kv put secret/myapp/env DEMO_API_KEY="supersecret" DEMO_ENV="production"

3. ⚙️ Kubernetes Resource Setup

Apply all manifests in sequence:

kubectl apply -f k8s/00-namespace.yaml
kubectl apply -f k8s/01-serviceaccount.yaml
kubectl apply -f k8s/02-secretstore.yaml
kubectl apply -f k8s/03-externalsecret.yaml
kubectl apply -f k8s/04-nginx-deployment.yaml

✅ Breakdown of Key Resources

`00-namespace.yaml`

apiVersion: v1
kind: Namespace
metadata:
  name: default

`01-serviceaccount.yaml`

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-sa
  namespace: default

`02-secretstore.yaml`

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: default
spec:
  provider:
    vault:
      server: "http://vault.vault.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "myapp-role"
          serviceAccountRef:
            name: external-secrets-sa

`03-externalsecret.yaml`

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: myapp-env-secret
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: secret/data/myapp/env

`04-nginx-deployment.yaml`

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo
    spec:
      serviceAccountName: external-secrets-sa
      containers:
        - name: nginx
          image: nginx:alpine
          envFrom:
            - secretRef:
                name: myapp-env-secret
          command: ["/bin/sh", "-c"]
          args:
            - echo "ENV VARS:"; env | grep DEMO_ ; sleep 3600

✅ Validation

Once the pod is up and running, verify that secrets have been correctly injected as environment variables:

kubectl get pods
kubectl exec -it <nginx-demo-pod> -- env | grep DEMO_

Expected Output:

DEMO_API_KEY=supersecret
DEMO_ENV=production

📎 Notes & Recommendations

Security: This guide uses Vault in dev mode and HTTP—not recommended for production. Use TLS and HA mode in production environments.
Policy Design: Keep policies granular and service-account specific to follow the principle of least privilege.
Refresh Behavior: ExternalSecrets will update Kubernetes secrets every hour (as per refreshInterval), ensuring changes in Vault propagate automatically.

Setup Vault in PROD persistent mode

To preserve Vault data across pod restarts or deletions, you need to move away from using Vault in dev mode, because:

❗ Vault in dev mode stores all data in-memory, so all secrets are lost when the pod is deleted or restarted.

✅ Solution: Use Vault in Production Mode with Persistent Storage

Here’s how you can update your setup so Vault retains secrets after pod restarts:

🔁 Step-by-Step Setup for Persistent Vault

1. Install Vault via Helm with Persistent Storage

Use the official HashiCorp Vault Helm chart (opens in a new tab) and enable persistent storage.

Add the repo and install:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create a `vault-values.yaml` with persistence enabled:

server:
  ha:
    enabled: false
  dev:
    enabled: false  # ⚠️ Disable dev mode for persistence
 
  dataStorage:
    enabled: true
    size: 1Gi
    storageClass: "standard"  # Use your cluster's appropriate storage class

Install Vault:

helm install vault hashicorp/vault -n vault --create-namespace -f vault-values.yaml

2. Initialize & Unseal Vault (only once)

kubectl exec -n vault -it vault-0 -- vault operator init

Store the unseal keys and root token securely.

Unseal the Vault manually or automate using auto-unseal with KMS (e.g., AWS KMS, Azure Key Vault, etc.) for production.

kubectl exec -n vault -it vault-0 -- vault operator unseal <key1>
kubectl exec -n vault -it vault-0 -- vault operator unseal <key2>
kubectl exec -n vault -it vault-0 -- vault operator unseal <key3>

3. Port Forward and Login to Vault

kubectl port-forward svc/vault -n vault 8200:8200
export VAULT_ADDR='http://127.0.0.1:8200'
vault login <root_token>

4. Continue with Your Existing Setup

Once you're using persistent Vault, the rest of your ESO integration stays mostly the same:

Kubernetes auth config
Secret paths like secret/data/myapp/env
ESO's SecretStore, ExternalSecret, etc.

✅ Now, even if the Vault pod restarts or is rescheduled, the secrets and configs will persist via the PVC.

🔒 Optional: Auto-Unseal Setup (Recommended for Production)

For HA or production-grade deployments, configure Vault auto-unseal via:

AWS KMS
Azure Key Vault
Google Cloud KMS

This avoids needing manual unseals on every restart.

📦 Validate PVC Attachment

Check that a PVC is bound:

kubectl get pvc -n vault

🧠 Summary

Setup Part	Dev Mode	Production Mode (Recommended)
Secrets persistence	❌ Lost on restart	✅ Persisted with PVC
Security	❌ Minimal (no TLS)	✅ Can be hardened
Use for demos	✅ Yes	✅ Yes (better choice)
Real-world readiness	❌ No	✅ Yes

Redpanda Console Alb Ingress