OpenBao Restore Using MinIO

This guide explains how to configure and restore Connect OpenBao using backups stored in MinIO.

For instructions on configuring OpenBao backups, see OpenBao Backup Guide.

Overview

Restoring OpenBao recovers the entire cluster from a full Raft snapshot stored in S3 or MinIO. Because OpenBao backups are full snapshots, not incremental backups, restoring always replaces the entire cluster state with the chosen snapshot. Use this process for disaster recovery or to roll back to a previous state after data corruption.

This operation replaces all existing data in the OpenBao cluster with the contents of the backup snapshot. Verify the backup timestamp and unseal keys before proceeding.

Configurable options:

  • Restore mode: Restore from the latest backup or from a specific snapshot by timestamp.

  • Unseal keys secret: Kubernetes secret that contains the unseal keys required to unseal OpenBao after restore.

  • S3 or MinIO configuration: S3 or MinIO endpoint, bucket, credentials, and TLS settings.

Configuration files:

For additional information on how to apply Helm value overrides, see Chart value override recommendations.

Compatibility:

  • MinIO: Fully tested and supported.

  • AWS S3: Expected to work with standard S3 APIs, but not yet validated in production.

Restored components

  • All secrets and key-value pairs.

  • Authentication methods and policies.

  • Secret engine configurations.

  • Audit device configurations.

  • System configuration and metadata.

Before you restore in a production environment, validate the complete backup and restore workflow in a test environment. For detailed testing procedures, see Test the Restore Process.

Prerequisites

  • Kubernetes cluster access with kubectl configured and the appropriate permissions to:

    • Access the namespace where connect-openbao is deployed.

    • Create Kubernetes secrets (required for restoring unseal keys).

  • Kubernetes secret for unseal keys. See Prepare Required Kubernetes Secrets for details.

  • Kubernetes secret for MinIO credentials. See Prepare Required Kubernetes Secrets for details.

  • Access to MinIO to fetch backup snapshots for Point-in-Time Restore and verify unseal keys:

    • Navigate to your MinIO user interface (for example, https://admin.<your-domain-name>/minio/console).

    • Sign in using your MinIO credentials.

    • Navigate to foundation-pf bucket → openbao-backup → snapshots/.

    • View snapshots: You will see files such as:

      • 2025-11-06_14-00-00-connect-openbao.snap

      • 2025-11-07_13-10-12-connect-openbao.snap

      • 2025-11-08_09-30-45-connect-openbao.snap

    • Note the timestamp. Copy the timestamp prefix, for example, 2025-11-07_13-10-12, for your restore configuration.

    • Download the unseal key reference file: Navigate to foundation-pf bucket → openbao-backup → unseal-keys/.

      • Download the file that matches your chosen snapshot timestamp, for example: 2025-11-07_13-10-12-connect-openbao-unseal-keys.txt.

      • Compare the first unseal key in the downloaded file with the unseal keys you backed up in Unseal Keys Backup to ensure they match.

      • This verification confirms that you have the correct unseal keys for the snapshot you are restoring.

Restore Configuration

Prepare Required Kubernetes Secrets

Create the unseal keys secret from your manual backup of unseal keys.

kubectl create secret generic connect-openbao-unseal-login-restore \
  --from-file=openbao.secret.txt=unseal-login.txt \
  -n foundation-env-default

Create the MinIO credentials secret by using one of the following methods:

Option 1: Run the following command to apply a SecretManager resource that copies MinIO credentials from the foundation-cluster-zerotrust namespace to the foundation-env-default namespace:

cat <<'EOF' | kubectl apply -f -
apiVersion: secretoperator.security.grid.digital.ge.com/v1beta1
kind: SecretManager
metadata:
  name: connect-openbao-restore
  namespace: foundation-env-default
spec:
  secrets:
  - name: minio1-secret
    namespace: foundation-cluster-zerotrust
    template:
      key: accesskey
      name: connect-openbao-minio1-creds-restore
      namespace: foundation-env-default
      value: '{{ index . "accesskey" }}'
  - name: minio1-secret
    namespace: foundation-cluster-zerotrust
    template:
      key: secretkey
      name: connect-openbao-minio1-creds-restore
      namespace: foundation-env-default
      value: '{{ index . "secretkey" }}'
EOF

Option 2: Create the secret directly if you have the credentials:

kubectl create secret generic connect-openbao-minio1-creds-restore \
  --from-literal=accesskey='your-access-key' \
  --from-literal=secretkey='your-secret-key' \
  -n foundation-env-default

S3 Common Configuration

The following configuration applies to restore operations that use MinIO or S3. Set all parameters exactly as specified for connectivity to work correctly.

  • s3CommonConfig.endpoint: S3/MinIO endpoint URL.

  • s3CommonConfig.region: S3/MinIO region.

  • s3CommonConfig.bucket: Target S3/MinIO bucket containing backups.

  • s3CommonConfig.auth.accessKey: Access key for authentication.

  • s3CommonConfig.auth.secretKey: Secret key for authentication.

  • s3CommonConfig.externalCa.enabled: Set to true if using a custom CA certificate.

  • s3CommonConfig.externalCa.cert: Certificate file name (for example, ca.crt).

  • s3CommonConfig.auth.secretName and s3CommonConfig.externalCa.secretName: Required if you provide your own Kubernetes secret (instead of relying on the SecretManager-generated one).

  • s3CommonConfig.forceAwsStyle: Set to true when using MinIO.

openbao:
  s3CommonConfig:
    endpoint: "https://minio.foundation-cluster-zerotrust:443"  # MinIO endpoint URL (or S3 endpoint for AWS)
    region: "us-east-1"  # Region name (use actual AWS region for S3)
    bucket: "foundation-pf"  # Bucket name
    auth:
      # secretName: minio1-creds  # Kubernetes secret containing the credentials. Leave this commented out if you are using the SecretManager block as specified in the prerequisites.
      accessKey: accesskey  # Access key
      secretKey: secretkey  # Secret key
    externalCa:
      enabled: true  # Enable custom CA for TLS
      # secretName: minio1-tls  # Kubernetes secret containing the TLS certificate. Leave this commented out if you are using the SecretManager block as specified in the prerequisites.
      cert: ca.crt  # Certificate file name
    forceAwsStyle: "true"  # Set to 'true' for MinIO, 'false' for AWS S3

S3 Restore Configuration

The following configuration is required to enable restore for an OpenBao cluster:

  • s3RestoreConfig.enabled: Set to true to enable restore.

  • s3RestoreConfig.timestamp: Optional. Specify a snapshot timestamp for point-in-time restore (for example, "2025-11-07_13-10-12"). Leave this value empty ("") to restore from the latest backup.

  • s3RestoreConfig.unsealKeysSecret: The name of the Kubernetes secret that contains the unseal keys from your manual backup.

  • server.extraInitContainers: Required. Configure the snapshot-download init container to download the snapshot from S3 or MinIO before OpenBao starts.

Option A: Restore from Latest Backup

openbao:
  server:
    extraInitContainers:
    - name: snapshot-download
      image: docker.io/amazon/aws-cli:2.31.39
      command: ['/bin/sh', '/openbao/snapshot-download/snapshot-download.sh']
      env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: accesskey
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: secretkey
      volumeMounts:
      - name: snapshot-download
        mountPath: /openbao/snapshot-download
        readOnly: true
      - name: restore
        mountPath: /openbao/restore
      - name: tmp
        mountPath: /openbao/tmp

  s3RestoreConfig:
    enabled: true  # Enable restore
    timestamp: ""  # Leave empty for latest backup
    unsealKeysSecret: "connect-openbao-unseal-login-restore"

Option B: Point-in-Time Restore (Timestamp Prefix)

openbao:
  server:
    extraInitContainers:
    - name: snapshot-download
      image: docker.io/amazon/aws-cli:2.31.39
      command: ['/bin/sh', '/openbao/snapshot-download/snapshot-download.sh']
      env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: accesskey
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: connect-openbao-minio1-creds-restore
            key: secretkey
      volumeMounts:
      - name: snapshot-download
        mountPath: /openbao/snapshot-download
        readOnly: true
      - name: restore
        mountPath: /openbao/restore
      - name: tmp
        mountPath: /openbao/tmp

  s3RestoreConfig:
    enabled: true  # Enable restore
    timestamp: "2025-11-07_13-10-12"  # Timestamp prefix
    unsealKeysSecret: "connect-openbao-unseal-login-restore"

Restore Procedure

Cleanup Existing Deployment

If OpenBao is already deployed, delete the existing release and its persistent volume claims (PVCs) before you restore.

 
# Uninstall the Helm release
helm uninstall connect-openbao -n foundation-env-default

# Delete PVCs and the old unseal secret
kubectl delete -n foundation-env-default \
  secret/connect-openbao-unseal-login \
  pvc/data-connect-openbao-{0..2} \
  pvc/audit-connect-openbao-{0..2}

Turn Off Backups During Restore

Temporarily turn off backups in the connect-openbao/values.yaml file to avoid conflicts during restore:

openbao:
  s3BackupConfig:
    enabled: false

Apply Restore Configuration

Add the S3 common configuration and S3 restore configuration to the connect-openbao/values.yaml file, and then deploy the release to perform the restore.

For guidance on applying Helm value overrides, see Chart value override recommendations.

Deploy or update the connect-openbao release with the updated values:

helm upgrade --install connect-openbao connect-helm/connect-openbao \
  --namespace foundation-env-default \
  --values connect-openbao/values.yaml

Verify Restore

Check pod status:

# All pods should be running
kubectl get pods -n foundation-env-default -l app.kubernetes.io/name=openbao

Check restore logs:

Check the logs for successful completion messages to confirm that the restore operation finished without errors.

# View init container logs (snapshot download)
# Look for: "Successfully downloaded snapshot"
kubectl logs connect-openbao-0 -c snapshot-download -n foundation-env-default

# View shared log file from inside pod
kubectl exec connect-openbao-0 -n foundation-env-default -- cat /openbao/tmp/post_start.log | grep -E "(init-snapshot-download|Restore from Backup)"

Verify OpenBao secrets:

Port-forward to localhost:8200 to access the OpenBao UI. Then sign in with your admin credentials and verify that secrets, policies, and configurations were restored correctly.

# Port-forward to OpenBao
kubectl port-forward svc/connect-openbao 8200:8200 -n foundation-env-default

Verify Connect Console:

In the Connect Console, select Security  Service Accounts to verify that existing service accounts are present and accessible. This confirms that OpenBao authentication and authorization mechanisms were restored correctly.

Post-Restore Configuration

Once verification is complete, update your values.yaml file to turn off restore mode and return to normal operations:

  1. Set openbao.s3RestoreConfig.enabled to false.

  2. Remove or comment out the snapshot-download init container from openbao.server.extraInitContainers.

  3. Turn on backups by setting openbao.s3BackupConfig.enabled to true.

openbao:
  s3RestoreConfig:
    enabled: false  # Disable restore mode
    timestamp: ""
    unsealKeysSecret: ""

  server:
    extraInitContainers: []  # Remove snapshot-download init container

Redeploy the chart:

# Redeploy without restore enabled
helm upgrade --install connect-openbao connect-helm/connect-openbao \
  --namespace foundation-env-default \
  --values connect-openbao/values.yaml

Cleanup Restore Resources

After you complete the restore and verification, remove the temporary restore resources.

# Delete the SecretManager for restore credentials (if used)
kubectl delete secretmanagers.secretoperator.security.grid.digital.ge.com connect-openbao-restore -n foundation-env-default

# Delete the restore secrets
kubectl delete secret connect-openbao-unseal-login-restore -n foundation-env-default
kubectl delete secret connect-openbao-minio1-creds-restore -n foundation-env-default

Test the Restore Process

For testing, validate the full backup and restore workflow in a test environment before you run it in production.

If you already have backup snapshots, use them for testing instead of creating new ones:

  • Option 1: Download backup files from your production MinIO instance and upload them to the test environment’s MinIO bucket at the same path (for example, foundation-pf/openbao-backup/snapshots/ and foundation-pf/openbao-backup/unseal-keys/).

  • Option 2: Configure the test environment to point directly to the MinIO instance that contains the production backups. Make sure network connectivity and credentials are configured correctly.

When you use existing backups, skip the Create test data, Trigger a manual backup, and Verify backup completion steps that follow. Go directly to Perform the restore by using an existing backup timestamp.

Test Environment Setup

  1. Deploy OpenBao with backups enabled. To configure and deploy OpenBao with backups, see OpenBao Backup Guide.

  2. Create test data:

  3. Trigger a manual backup: Trigger a backup job immediately, rather than waiting for the scheduled CronJob to run.

    # Create a one-time job from the backup CronJob
kubectl create job --from=cronjob/connect-openbao-backup connect-openbao-backup-test \
  -n foundation-env-default

  4. Verify backup completion: Follow the verification steps in Backup Verification to confirm the snapshot was created successfully in MinIO.

  5. Perform the restore:

    • Follow the Restore Configuration and Restore Procedure steps to restore from the backup.

    • Use the timestamp of the backup you just created (or an existing backup if you used the alternative approach above).

  6. Verify restored data: Follow the Verify Restore steps to confirm that all OpenBao data was restored successfully.