OpenBao Restore Using MinIO

This guide explains how to configure and restore Connect OpenBao using backups stored in MinIO.

For instructions on configuring OpenBao backups, see OpenBao Backup Guide.

Overview

Restoring OpenBao recovers the entire cluster from a Raft snapshot stored in S3 or MinIO. This process is critical for disaster recovery or for rolling back to a previous state after data corruption.

This operation replaces all existing data in the OpenBao cluster with the contents of the backup snapshot. Verify the backup timestamp and unseal keys before proceeding.

Compatibility:

  • MinIO: Fully tested and supported.

  • AWS S3: Expected to work with standard S3 APIs (not yet validated in production).

Restored components

  • All secrets and key-value pairs.

  • Authentication methods and policies.

  • Secret engine configurations.

  • Audit device configurations.

  • System configuration and metadata.

Before you restore in a production environment, validate the complete backup and restore workflow in a test environment. For detailed testing procedures, see Test the Restore Process.

Prerequisites

  • Kubernetes cluster access with kubectl configured and appropriate permissions to:

    • Access the namespace where connect-openbao is deployed.

    • Create Kubernetes secrets (required for restoring unseal keys).

  • Kubernetes Secret for unseal keys. See Prepare Required Kubernetes Secrets for details.

  • Kubernetes Secret for MinIO credentials. See Prepare Required Kubernetes Secrets for details.

  • Access to MinIO to fetch backup snapshots for Point-in-Time Restore and verify unseal keys:

    • Navigate to your MinIO user interface (for example, https://admin.<your-domain-name>/minio/console).

    • Sign in using your MinIO credentials.

    • Navigate to foundation-pf bucket → openbao-backup → snapshots/.

    • View snapshots: You will see files such as:

      • 2025-11-06_14-00-00-connect-openbao.snap

      • 2025-11-07_13-10-12-connect-openbao.snap

      • 2025-11-08_09-30-45-connect-openbao.snap

    • Note the timestamp: Copy the timestamp prefix (for example, 2025-11-07_13-10-12) for your restore configuration.

    • Download unseal keys reference: Navigate to foundation-pf bucket → openbao-backup → unseal-keys/.

      • Download the file that matches your chosen snapshot timestamp, for example: 2025-11-07_13-10-12-connect-openbao-unseal-keys.txt.

      • Compare the first unseal key in the downloaded file with the unseal keys you backed up in Unseal Keys Backup to ensure they match.

      • This verification confirms you have the correct unseal keys for the snapshot you are restoring.

Restore Configuration

Prepare Required Kubernetes Secrets

Create unseal keys secret from your manual backup of unseal keys.

kubectl create secret generic connect-openbao-unseal-login-restore \
  --from-file=openbao.secret.txt=unseal-login.txt \
  -n foundation-env-default

Create the MinIO credentials secret using one of the following methods:

Option 1: Copy the credentials from foundation-cluster-zerotrust namespace to the foundation-env-default namespace using SecretManager:

cat <<'EOF' | kubectl apply -f -
apiVersion: secretoperator.security.grid.digital.ge.com/v1beta1
kind: SecretManager
metadata:
  name: connect-openbao-restore
  namespace: foundation-env-default
spec:
  secrets:
  - name: minio1-secret
    namespace: foundation-cluster-zerotrust
    template:
      key: accesskey
      name: connect-openbao-minio1-creds-restore
      namespace: foundation-env-default
      value: '{{ index . "accesskey" }}'
  - name: minio1-secret
    namespace: foundation-cluster-zerotrust
    template:
      key: secretkey
      name: connect-openbao-minio1-creds-restore
      namespace: foundation-env-default
      value: '{{ index . "secretkey" }}'
EOF

Option 2: Create the secret directly if you have the credentials:

kubectl create secret generic connect-openbao-minio1-creds-restore \
  --from-literal=accesskey='your-access-key' \
  --from-literal=secretkey='your-secret-key' \
  -n foundation-env-default

S3 Common Configuration

The following configuration applies to restore operations using MinIO or S3. All parameters must be set exactly as specified for connectivity to work correctly.

  • s3CommonConfig.endpoint: S3/MinIO endpoint URL.

  • s3CommonConfig.region: S3/MinIO region.

  • s3CommonConfig.bucket: Target S3/MinIO bucket containing backups.

  • s3CommonConfig.auth.accessKey: Access key for authentication.

  • s3CommonConfig.auth.secretKey: Secret key for authentication.

  • s3CommonConfig.externalCa.enabled: Set to true if using a custom CA certificate.

  • s3CommonConfig.externalCa.cert: Certificate file name (for example, ca.crt).

  • s3CommonConfig.auth.secretName and s3CommonConfig.externalCa.secretName: Required if you provide your own Kubernetes secret (instead of relying on the SecretManager-generated one).

  • s3CommonConfig.forceAwsStyle: Set to true when using MinIO.

openbao:
  s3CommonConfig:
    endpoint: "https://minio.foundation-cluster-zerotrust:443"  # MinIO endpoint URL (or S3 endpoint for AWS)
    region: "us-east-1"  # Region name (use actual AWS region for S3)
    bucket: "foundation-pf"  # Bucket name
    auth:
      # secretName: minio1-creds  # Kubernetes secret containing the credentials. Leave this commented out if you are using the SecretManager block as specified in the prerequisites.
      accessKey: accesskey  # Access key
      secretKey: secretkey  # Secret key
    externalCa:
      enabled: true  # Enable custom CA for TLS
      # secretName: minio1-tls  # Kubernetes secret containing the TLS certificate. Leave this commented out if you are using the SecretManager block as specified in the prerequisites.
      cert: ca.crt  # Certificate file name
    forceAwsStyle: "true"  # Set to 'true' for MinIO, 'false' for AWS S3

S3 Restore Configuration

The following configuration is required to enable restore for an OpenBao cluster:

  • s3RestoreConfig.enabled: Set to true to enable restore.

  • s3RestoreConfig.timestamp: Optional. Specify a snapshot timestamp for point-in-time restore (for example, "2025-11-07_13-10-12"). Leave empty ("") to restore from the latest backup.

  • s3RestoreConfig.unsealKeysSecret: The name of the Kubernetes secret that contains the unseal keys from your manual backup.

  • server.extraInitContainers: Required. Configure the snapshot-download init container to download the snapshot from S3 or MinIO before OpenBao starts.

Option A: Restore from Latest Backup

openbao:
  server:
    extraInitContainers:
    - name: snapshot-download
      image: docker.io/amazon/aws-cli:2.31.39
      command: ['/bin/sh', '/openbao/snapshot-download/snapshot-download.sh']
      env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: accesskey
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: secretkey
      volumeMounts:
      - name: snapshot-download
        mountPath: /openbao/snapshot-download
        readOnly: true
      - name: restore
        mountPath: /openbao/restore
      - name: tmp
        mountPath: /openbao/tmp

  s3RestoreConfig:
    enabled: true  # Enable restore
    timestamp: ""  # Leave empty for latest backup
    unsealKeysSecret: "connect-openbao-unseal-login-restore"

Option B: Point-in-Time Restore (Timestamp Prefix)

openbao:
  server:
    extraInitContainers:
    - name: snapshot-download
      image: docker.io/amazon/aws-cli:2.31.39
      command: ['/bin/sh', '/openbao/snapshot-download/snapshot-download.sh']
      env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: accesskey
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: secretkey
      volumeMounts:
      - name: snapshot-download
        mountPath: /openbao/snapshot-download
        readOnly: true
      - name: restore
        mountPath: /openbao/restore
      - name: tmp
        mountPath: /openbao/tmp

  s3RestoreConfig:
    enabled: true  # Enable restore
    timestamp: "2025-11-07_13-10-12"  # Timestamp prefix
    unsealKeysSecret: "connect-openbao-unseal-login-restore"

Restore Procedure

Cleanup Existing Deployment

If OpenBao is already deployed, delete the existing release and its persistent volume claims (PVCs) before you restore.

 
# Uninstall the Helm release
helm uninstall connect-openbao -n foundation-env-default

# Delete PVCs and the old unseal secret
kubectl delete -n foundation-env-default \
  secret/connect-openbao-unseal-login \
  pvc/data-connect-openbao-{0..2} \
  pvc/audit-connect-openbao-{0..2}

Turn Off Backups During Restore

Temporarily turn off backups in the connect-openbao/values.yaml file to avoid conflicts during the restore process:

openbao:
  s3BackupConfig:
    enabled: false

Apply Restore Configuration

Add the S3 common configuration and S3 restore configuration to the values.yaml file, and then deploy the connect-openbao release to perform the restore.

# Deploy with restore configuration
helm upgrade --install connect-openbao . \
  --namespace foundation-env-default \
  --values values.yaml

Verify Restore

Check pod status:

# All pods should be running
kubectl get pods -n foundation-env-default -l app.kubernetes.io/name=openbao

Check restore logs:

Check the logs for successful completion messages to confirm that the restore operation finished without errors.

# View init container logs (snapshot download)
# Look for: "Successfully downloaded snapshot"
kubectl logs connect-openbao-0 -c snapshot-download -n foundation-env-default

# View shared log file from inside pod
kubectl exec connect-openbao-0 -n foundation-env-default -- cat /openbao/tmp/post_start.log | grep -E "(init-snapshot-download|Restore from Backup)"

Verify OpenBao secrets:

Port-forward to localhost:8200 to access the OpenBao UI. Then sign in with your admin credentials and verify that secrets, policies, and configurations are restored correctly.

# Port-forward to OpenBao
kubectl port-forward svc/connect-openbao 8200:8200 -n foundation-env-default

Verify Connect Console:

Navigate to the Connect Console, and select the Service Account to verify that the existing service accounts are present and accessible. This confirms that OpenBao’s authentication and authorization mechanisms have been restored correctly.

Post-Restore Configuration

Once the verification is complete, update your values.yaml to turn off restore mode and return to normal operations:

  1. Set openbao.s3RestoreConfig.enabled to false.

  2. Remove or comment out the snapshot-download init container from openbao.server.extraInitContainers.

  3. Turn on backups by setting openbao.s3BackupConfig.enabled to true.

openbao:
  s3RestoreConfig:
    enabled: false  # Disable restore mode
    timestamp: ""
    unsealKeysSecret: ""

  server:
    extraInitContainers: []  # Remove snapshot-download init container

Redeploy the chart:

# Redeploy without restore enabled
helm upgrade connect-openbao . -n foundation-env-default --values values.yaml

Cleanup Restore Resources

After you complete the restore and verification, remove the temporary restore resources.

# Delete the SecretManager for restore credentials (if used)
kubectl delete secretmanagers.secretoperator.security.grid.digital.ge.com connect-openbao-restore -n foundation-env-default

# Delete the restore secrets
kubectl delete secret connect-openbao-unseal-login-restore -n foundation-env-default
kubectl delete secret connect-openbao-minio1-creds-restore -n foundation-env-default

Test the Restore Process

For testing, validate the full backup and restore workflow in a test environment before you run it in production.

If you already have existing backup snapshots, use them for testing instead of creating new ones:

  • Option 1: Download backup files from your production MinIO instance and upload them to the test environment’s MinIO bucket at the same path (for example, foundation-pf/openbao-backup/snapshots/ and foundation-pf/openbao-backup/unseal-keys/).

  • Option 2: Configure the test environment to point directly to the MinIO instance that contains the production backups. Make sure that network connectivity and credentials are configured correctly.

When you use existing backups, skip the Create test data, Trigger a manual backup, and Verify backup completion steps below. Go directly to Perform the restore using an existing backup timestamp.

Test Environment Setup

  1. Deploy OpenBao with backups enabled. To configure and deploy OpenBao with backups, see OpenBao Backup Guide.

  2. Create test data:

  3. Trigger a manual backup: Trigger a backup job immediately, rather than waiting for the scheduled CronJob to run.

    # Create a one-time job from the backup CronJob
kubectl create job --from=cronjob/connect-openbao-backup connect-openbao-backup-test \
  -n foundation-env-default

  4. Verify backup completion: Follow the verification steps in Backup Verification to confirm the snapshot was created successfully in MinIO.

  5. Perform the restore:

    • Follow the Restore Configuration and Restore Procedure steps to restore from the backup.

    • Use the timestamp of the backup you just created (or an existing backup if you used the alternative approach above).

  6. Verify restored data: Follow the Verify Restore steps to confirm that all OpenBao data was restored successfully.