Connect - External PostgreSQL

This guide covers deploying Utilihive with an externally managed PostgreSQL server instead of the in-cluster Zalando operator. Any standard PostgreSQL server that is reachable from the cluster can work, provided you have the correct credentials and connectivity.

Verified backends:

  • Azure PostgreSQL Flexible Server

  • AWS RDS for PostgreSQL

  • GCP Cloud SQL for PostgreSQL

The databaseProvisioning and database.provider: external blocks handle database creation, user provisioning, and Kubernetes Secret management automatically through Helm pre-install and pre-upgrade hooks.

Overview

Component Role

External PostgreSQL server

Managed relational database (Azure Flexible Server, AWS RDS, or other). One shared server, one logical database per Connect service.

Admin Secret (connect-postgresql-external-admin)

Pre-created Kubernetes Secret that stores the server admin credentials. Used to create databases and users for Connect services. Not managed by Helm.

CA Secret (connect-postgresql-external-ca)

Pre-created Kubernetes Secret that stores the CA certificate bundle used to verify TLS. Contents differ by provider — see Step 2: Create a Kubernetes Secret containing the CA certificate. Not managed by Helm.

Connect values override

External database configuration in connect/values.yaml (for example, database.provider: external, database.external.host, database.external.existingSecret, and database.external.secretsTemplates).

Example Configurations

Reference override files are provided in the examples/ directory:

Prerequisites

Step 1: Create a Kubernetes Secret containing external PostgreSQL admin credentials

The provisioning Job needs server-level credentials to create databases and users for Connect services. Store these in a Kubernetes Secret named connect-postgresql-external-admin in the same namespace as the Connect release (for example, foundation-env-default).

kubectl create secret generic connect-postgresql-external-admin \
  --from-literal=POSTGRES_HOST=<server-fqdn-or-endpoint> \
  --from-literal=POSTGRES_PORT=5432 \
  --from-literal=POSTGRES_USER=<admin-username> \
  --from-literal=POSTGRES_PASSWORD=<admin-password> \
  -n foundation-env-default

Replace <server-fqdn-or-endpoint>, <admin-username>, and <admin-password> with your server’s values.

POSTGRES_PORT is optional and defaults to 5432.

Step 2: Create a Kubernetes Secret containing the CA certificate

All supported providers enforce TLS by default. The CA certificate is required for sslMode: verify-ca or verify-full (certificate verification). Without it, you can still connect with sslMode: require (encrypted, no certificate verification), but do not use this mode in production.

The Secret key must be ca.crt, it is mounted at /etc/secrets/ca/postgresql.crt in each service pod.

Azure PostgreSQL Flexible Server

Refer to Microsoft documentation for more information.

  1. Download the CA certificates:

    curl -L -o DigiCertGlobalRootG2.crt https://cacerts.digicert.com/DigiCertGlobalRootG2.crt
curl -L -o MicrosoftRSA2017.crt https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt

  2. Convert from DER to PEM:

    openssl x509 -inform DER -in DigiCertGlobalRootG2.crt -out DigiCertGlobalRootG2.pem
openssl x509 -inform DER -in MicrosoftRSA2017.crt -out MicrosoftRSA2017.pem

  3. Combine into a single bundle:

    cat DigiCertGlobalRootG2.pem MicrosoftRSA2017.pem > azure-pg-ca-bundle.pem
grep -c "BEGIN CERTIFICATE" azure-pg-ca-bundle.pem  # expect: 2

  4. Create the Kubernetes Secret:

    kubectl create secret generic connect-postgresql-external-ca \
  --from-file=ca.crt=azure-pg-ca-bundle.pem \
  -n foundation-env-default

AWS RDS for PostgreSQL

AWS publishes a single global CA bundle covering all regions and generations. See AWS documentation for more information.

  1. Download the bundle:

    curl -L -o global-bundle.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
grep -c "BEGIN CERTIFICATE" global-bundle.pem  # expect: many (covers all regions/CAs)

  2. Create the Kubernetes Secret:

    kubectl create secret generic connect-postgresql-external-ca \
  --from-file=ca.crt=global-bundle.pem \
  -n foundation-env-default

GCP Cloud SQL for PostgreSQL

GCP Cloud SQL uses a per-instance self-signed CA, there is no shared public bundle. Download the server CA cert via gcloud or from the Cloud Console.

  1. Download the CA certificate:

    gcloud sql instances describe <INSTANCE_NAME> \
  --format="value(serverCaCert.cert)" > server-ca.pem

    Or from Cloud Console: Cloud SQL instance → Connections → Security → Download server certificate.

  2. Create the Kubernetes Secret:

    kubectl create secret generic connect-postgresql-external-ca \
  --from-file=ca.crt=server-ca.pem \
  -n foundation-env-default

Cloud SQL TLS quirks:

  • The server cert SAN is the instance’s internal DNS name (for example, 2-xxxx.europe-west1.sql.goog), not the IP address or FQDN you connect to. Use sslMode: verify-ca (not verify-full) for Java/JDBC services. verify-full checks the hostname and fails for this certificate pattern.

  • The Node.js pg library used by frontendconfiguration does not honor sslmode=verify-ca semantics, it always checks the hostname. Set PGSSLMODE: no-verify in secretsTemplates block to enable SSL without hostname verification (see examples/values-external-postgres-gcp.yaml).

Step 3: Configure connect/values.yaml

In your environment connect/values.yaml, update overrides to be similar to example files in the examples/ directory. Key fields per service:

databaseProvisioning:
  enabled: true
  adminSecretName: connect-postgresql-external-admin  # Secret from Step 1

global:
  postgresql:
    caSecretName: connect-postgresql-external-ca      # Secret from Step 2

<service>:
  database:
    provider: external
    external:
      host: <server-fqdn>
      database: <db-name>
      username: <db-user>
      sslMode: verify-full # or verify-ca, see note below
      existingSecret: connect-<service>-external-db-creds
      secretsTemplates:
        CONNECTION_STRING: "jdbc:{{ .Protocol }}://{{ .DatabaseHost }}:{{ .DatabasePort }}/{{ .DatabaseName }}?ssl=true&sslmode={{ .SslMode }}&sslrootcert=/etc/secrets/ca/postgresql.crt"
sslMode: verify-ca vs verify-full

  • verify-ca (default) — validates the server cert against the CA bundle but does not check the hostname. Suitable for Azure PostgreSQL Flexible Server and GCP Cloud SQL (Cloud SQL cert SAN is the internal DNS name, not the IP/FQDN).

  • verify-full — also validates that the hostname matches the cert CN/SAN. Required for AWS RDS (the RDS endpoint hostname matches the cert CN).

  • GCP Cloud SQL — frontendconfiguration (Node.js pg): sslMode is ignored for this service. Set PGSSLMODE: no-verify in secretsTemplates block. See GCP Cloud SQL for PostgreSQL for details.

What happens during deployment:

  1. provider: external switches the service from the in-cluster DB operator to the external server.

  2. existingSecret tells the chart not to render the database-external-secret.yaml template. The provisioning Job creates this Secret instead. Secrets are named <release>-<service>-external-db-creds (for example, connect-flowserver-external-db-creds). The -external-db-creds suffix avoids collision with the db-operator-managed <release>-<service>-db-creds secrets when upgrading from in-cluster PostgreSQL.

  3. secretsTemplates defines extra keys written into the credential Secret. Available placeholders: {{ .Protocol }}, {{ .DatabaseHost }}, {{ .DatabasePort }}, {{ .DatabaseName }}, {{ .UserName }}, {{ .Password }}, {{ .SslMode }}.

Step 4: Deploy

Follow the installation guide: Deploy Connect

What happens during deployment When database.provider: external is configured, the provisioning Job runs automatically as a pre-install/pre-upgrade hook, so databases and users are ready before any app pods start. For each service it:

  1. Connects to the external PostgreSQL server using the admin Secret.

  2. Creates databases (idempotent).

  3. Creates users (idempotent).

  4. Grants ALL PRIVILEGES ON DATABASE and USAGE, CREATE ON SCHEMA public (required for PostgreSQL 15+).

  5. Creates a per-service Kubernetes Secret, or reuses the existing password if the Secret already exists (safe on upgrade).

Verify the Deployment

Verify provisioning job logs

kubectl -n foundation-env-default logs -l job-name=connect-db-provision --tail=100

Example output:

{"level":"INFO","timeMillis":...,"message":"Connecting to <host>:5432 as <admin>"}
{"level":"INFO","timeMillis":...,"message":"Provisioning database 'flowserver' for user 'flowserver', secret: connect-flowserver-external-db-creds"}
{"level":"INFO","timeMillis":...,"message":"Database 'flowserver' already exists"}
{"level":"INFO","timeMillis":...,"message":"Granted USAGE, CREATE on schema public in database 'flowserver' to user 'flowserver'"}
...
{"level":"INFO","timeMillis":...,"message":"Provisioning complete."}

Inspect a credential secret

kubectl -n foundation-env-default get secret connect-flowserver-external-db-creds -o json \
  | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'

Verify all credential secrets

for svc in flowserver identity resourceregistry insights frontendconfiguration; do
  echo "=== $svc ==="
  kubectl -n foundation-env-default get secret connect-${svc}-external-db-creds -o json \
    | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'
done

Verify hook resources are cleaned up after install or upgrade

After a successful helm install or helm upgrade, all hook resources are automatically deleted (hook-delete-policy: hook-succeeded). Verify nothing remains:

kubectl get sa,role,rolebinding,cm,job -n foundation-env-default | grep "db-provision\|db-cleanup"

Expect no output.

Verify Connect pods are running

Verify all Connect service pods are up and ready:

kubectl get pods -n foundation-env-default -l app.kubernetes.io/instance=connect

All pods should show Running status with ready containers. If any pod is in CrashLoopBackOff or Error state, check its logs for database connectivity errors (see the Troubleshooting section).

After these checks pass, Connect should be running, and you should be able to access the Connect Console.

How to Uninstall

Refer to the standard uninstall guide: Uninstall Connect

When using an external database, running helm uninstall connect triggers the pre-delete hook, which runs the connect-db-cleanup Job before Helm removes the release. The cleanup Job deletes all per-service *-external-db-creds Secrets from the namespace. The admin Secret (connect-postgresql-external-admin) and CA Secret (connect-postgresql-external-ca) are *not* managed by Helm and are left intact.

To verify cleanup completed:

# Should return no results
kubectl get secret -n foundation-env-default | grep "external-db-creds"
# Hook resources should also be gone
kubectl get sa,role,rolebinding,cm,job -n foundation-env-default | grep "db-provision\|db-cleanup"

Generate and Rotate Passwords

Generate Passwords

On first install, the provisioning Job generates a 32-character secure random password for each service and stores it in the per-service Kubernetes Secret. Passwords meet common complexity requirements (uppercase, lowercase, digit, and a special character from !#%*+-=?_), making them compatible with providers that enforce password policies such as GCP Cloud SQL.

Rotate Passwords

The provisioning Job reuses the existing password on upgrade (it reads the current Secret before generating anything). To rotate a password:

  1. Delete the credential Secret for the target service:

    kubectl -n foundation-env-default delete secret connect-<service>-external-db-creds

  2. Run helm upgrade (or helmfile apply), the Job will generate a new password, update the PostgreSQL user, and recreate the Secret.

  3. The app pods will restart and pick up the new credentials automatically.

Troubleshooting

Symptom Likely cause / fix

Job fails: permission denied for schema public

PostgreSQL 15+ revoked CREATE on the public schema. Re-run the provisioning Job (helm upgrade), the Job now includes GRANT USAGE, CREATE ON SCHEMA public. Ensure the connect chart version includes this fix.

Job fails: could not connect to server: FATAL: password authentication failed

connect-postgresql-external-admin Secret has wrong credentials. Verify with kubectl get secret connect-postgresql-external-admin -o json | jq -r '.data | to_entries[] | "\(.key): \(.value | @base64d)"'.

Job fails: SSL SYSCALL error or certificate error

Wrong CA certificate in connect-postgresql-external-ca. The ca.crt key must contain the root CA that signed the server’s TLS certificate. See Step 2: Create a Kubernetes Secret containing the CA certificate for provider-specific download instructions.

App pods crash: permission denied for schema public

The provisioning Job ran before the schema grant fix was added. Delete the affected Secret(s), then redeploy the chart (helm upgrade) so the provisioning Job runs again with the grant fix included (see Generate and Rotate Passwords).

App pods crash: could not read certificate file

The caSecretName is wrong or the Secret doesn’t have a ca.crt key. Verify with kubectl -n foundation-env-default get secret connect-postgresql-external-ca.

GCP — frontendconfiguration crashes: ERR_TLS_CERT_ALTNAME_INVALID or pg_hba.conf rejects connection … no encryption

The Cloud SQL cert SAN is the instance’s internal DNS name. Node.js TLS rejects it, and fallback values (verify-ca, require, disable) either fail hostname check or send no SSL at all. Set PGSSLMODE: no-verify in frontendconfiguration.database.external.secretsTemplates (Node.js pg-specific value: enables SSL, skips hostname verification).