Troubleshooting

This section helps you identify, diagnose, and resolve common issues in GridOS Connect.

1. Access

  • If you have authentication problems when accessing services, or if your session has timed out, sign in to Zitadel with the credentials obtained via Access Platform Services and then force refresh your service web page.

  • If you have trouble accessing any services, make sure you delete all cookies and local storage/cache. Alternatively, use the incognito mode of your web browser.

2. Load Balancers and Reverse Proxies

You do not require any additional configuration for loadbalancers for Connect. As long as the loadbalancer points to the VMs port 443 and preserves the host in http requests (for application loadbalancers), and all required DNS names are added to point to the loadbalancer (zitadel, service, console, api), the load balancer will work.

3. Flow Traces Missing After Deployment or Upgrade

After you deploy or upgrade Connect, Fluent Bit might not collect flowserver logs automatically. The Fluent Bit Operator might not reload its configuration after Connect adds the clusterInput, clusterFilter, and clusterOutput custom resources to the cluster. During a fresh installation, restart the Fluent Bit and Flow Server pods after deployment to prevent this issue. For instructions, see Restart Fluent Bit and Flow Server Pods.

Symptom:

  • Flow traces are not visible in the Connect Console after a deployment or upgrade

  • An index_not_found_exception error is logged for the flowserver-logs index

Root Cause:

The Fluent Bit Operator does not automatically reload its pipeline configuration when Connect adds custom resources during installation or upgrade. Restart the Fluent Bit pods to reload the configuration. Then restart the Flow Server pods so they send logs to the updated pipeline.

Solution:

Restart Fluent Bit pods:

kubectl delete po -n foundation-cluster-monitoring -l app.kubernetes.io/name=fluent-bit

Restart Flow Server pods:

kubectl delete po -n foundation-env-default -l app.kubernetes.io/name=flowserver

The commands above use the default namespaces. If your environment uses different namespaces, replace the namespace value in each command with the appropriate namespace.

Verification:

After restarting the Fluent Bit and Flow Server pods, wait a few minutes for them to reach the Running state. Then trigger a flow in the Connect Console and verify that its traces appear. If they do not, check the Fluent Bit pod logs for errors:

kubectl logs -n foundation-cluster-monitoring -l app.kubernetes.io/name=fluent-bit

4. Prevent Log Duplication of Connect Flow Server

The default Foundation fluent-bit clusterOutput configuration leads to log duplication across Elasticsearch data streams. Each log event is stored multiple times, which increases storage usage and might impact query performance.

Specifically, for each log event indexed as a document in Elasticsearch from Connect flow-server or other Connect services, the following happens:

  • Two copies are stored in the logs data stream if the log event originates from any Connect service, including flow-server. This happens because the default Foundation fluent-bit configuration uses a catch-all clusterInput that collects logs from all containers, including Connect services, and sends them to the logs data stream in Elasticsearch.

  • One copy is stored in the flowserver-logs data stream if the log event originates from the Connect flow-server.

  • One copy is stored in the connect-services data stream if the log event originates from a Connect service other than flow-server.

As a result, each Connect-related log event can be stored up to three times.

Solution:
Since Connect flow-server logs only need to be retained in the flowserver-logs data stream, and other Connect services logs in the connect-services data stream, exclude these logs from the logs data stream. This reduces redundancy and optimizes storage.

The Foundation clusterInput exclusion prevents the catch-all Foundation log collector from reading Connect container logs, while the catch-all Elasticsearch clusterOutput update ensures Connect logs are not written to the default logs data stream.

Apply both changes below:

  • Exclude Connect logs from the default Foundation clusterInput:

    Ensure that the Foundation Helm overlay values have been used to redeploy the Foundation monitoring-apps Helm release in your environment. This configuration adds the connect- pattern to the extraExcludedContainers field of the catch-all Foundation clusterInput. This prevents log collection for all Connect containers, including flow-server, in the default logs data stream.

    ...
fluent-bit:
  enabled: true
  fluentBit:
    input:
      extraExcludedContainers: "*connect-*"
...

  • Update catch-all Elasticsearch clusterOutput:

    Assuming the clusteroutput.fluentbit.fluent.io/elasticsearch resource has not been modified outside from default Foundation configuration.

    1. Fetch elasticsearch clusterOutput configuration

      kubectl get clusteroutputs elasticsearch -o yaml

    2. Modify elasticsearch clusterOutput the configuration

      kubectl get clusteroutput elasticsearch -o yaml \
  | yq e '
      del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |
      del(.metadata.generation) |
      del(.metadata.resourceVersion) |
      del(.metadata.uid) |
      del(.metadata.creationTimestamp)
    ' - \
  | yq e '
      .spec |= (
        select(has("match")) |
        .matchRegex = "^(?!flowserver.*)(?!connect.*).*" |
        del(.match)
      )
    ' -

    3. Compare the outputs Compare and verify that the output from steps 1 and 2 matches the table below. The match field should be replaced with matchRegex.

      Original Modified 
      apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  annotations:
    meta.helm.sh/release-name: monitoring-apps
    meta.helm.sh/release-namespace: foundation-cluster-monitoring
  labels:
    app.kubernetes.io/managed-by: Helm
    fluentbit.fluent.io/enabled: "true"
  name: elasticsearch
spec:
  es:
    generateID: true
    host: monitoring-apps-es-client
    index: logs
    logstashFormat: false
    port: 9200
    replaceDots: true
    suppressTypeName: "On"
    timeKey: '@timestamp'
    traceError: false
    traceOutput: false
    type: _doc
  match: '*'
  retry_limit: "False"
      		 
      apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  annotations:
    meta.helm.sh/release-name: monitoring-apps
    meta.helm.sh/release-namespace: foundation-cluster-monitoring
  labels:
    app.kubernetes.io/managed-by: Helm
    fluentbit.fluent.io/enabled: "true"
  name: elasticsearch
spec:
  es:
    generateID: true
    host: monitoring-apps-es-client
    index: logs
    logstashFormat: false
    port: 9200
    replaceDots: true
    suppressTypeName: "On"
    timeKey: '@timestamp'
    traceError: false
    traceOutput: false
    type: _doc
  matchRegex: ^(?!flowserver.*)(?!connect.*).*
  retry_limit: "False"

      If your output is consistent with the table above, use the following commands to exclude Connect-related tags in the elasticsearch clusterOutput.

    4. Create a YAML file for the modified clusterOutput resource

      kubectl get clusteroutput elasticsearch -o yaml \
  | yq e '
      del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration") |
      del(.metadata.generation) |
      del(.metadata.resourceVersion) |
      del(.metadata.uid) |
      del(.metadata.creationTimestamp)
    ' - \
  | yq e '
      .spec |= (
        select(has("match")) |
        .matchRegex = "^(?!flowserver.*)(?!connect.*).*" |
        del(.match)
      )
    ' - \
  > modified-clusteroutput-es.yaml

    5. Delete the old clusterOutput resource

      kubectl delete clusteroutput elasticsearch

    6. Apply the modified clusterOutput resource

      kubectl apply -f modified-clusteroutput-es.yaml

    7. Restart fluent-bit pods

      kubectl delete po -n foundation-cluster-monitoring -l app.kubernetes.io/name=fluent-bit

5. Connect-PostgreSQL Fails to Install

If you are installing connect on a Foundation version lower than 25R01, the connect-postgresql Helm chart will fail with the following error:

Error: UPGRADE FAILED: cannot patch "connect-postgresql" with kind postgresql: postgresql.acid.zalan.do "connect-postgresql" is invalid: spec.postgresql.version: Unsupported value: "17": supported values: "10", "11", "12", "13", "14", "15"

Resolution Options:

  • Upgrade Foundation to 25R01+

Upgrade your environment to Foundation version 25R01 or higher and then run apply-helmfile to redeploy the GridOS connect-postgresql with the latest supported version.

  • Downgrade connect-postgresql chart

Edit helmfile.yaml.gotmpl in the root directory of the Helm Deployment Template.

  • Ensure that the connect-postgresql chart version is set to 0.5.0, and run apply-helmfile to redeploy the GridOS connect-postgresql.

...
- name: connect-postgresql
  <<: *defaults
  chart: connect-helm/connect-postgresql
  version: 0.5.0
  values:
  - ./connect-postgresql/values.yaml
  # - ./offline-image-overrides.yaml
...

6. connect-identityreconciler Pod Fails to Start Due to Renamed APISIX Secret

Connect release 1.20.0 includes updates to support changes in APISIX secrets introduced in Foundation version 25r09.

Symptom: The connect-identityreconciler pod may fail to start and log one of the following errors:

  • When deploying Connect from scratch:

    MountVolume.SetUp failed for volume "secrets": secret "connect-identityreconciler-apisix-api-token" not found

  • When upgrading Connect:

    Getting upstream 'connect-flowserver-service' failed with status: 401 Unauthorized

Root Cause: In Foundation version 25r09, an APISIX secret was renamed. Connect relies on this secret, so if Connect and Foundation are on mismatched versions, the secret name does not align. This can cause the pod to fail at startup or result in service requests being rejected.

Compatibility Overview: The following matrix shows the required actions based on the versions of Foundation and Connect:

Foundation version Connect version Action

25r09 or later

1.20.0 or later

No action required; the setup is fully compatible.

Older than 25r09

1.20.0 or later

Apply the override described in Solution.

25r09 or later

Older than 1.20.0

Apply the override described in Solution.

Solution:

In connect/values.yaml, add the following override under the identityreconciler block:

Foundation version older than 25r09 and Connect 1.20.0 or later

Foundation version 25r09 or later and Connect version older than 1.20.0

 
# Override for Foundation older than 25r09 and Connect 1.20.0+
secretManager:
  secrets:
    apisix-api-token:
      name: apisix-control-plane-api-token
 
# Override for Foundation 25r09+ and Connect version older than 1.20.0
secretManager:
  secrets:
    apisix-api-token:
      name: apisix-control-plane-token

7. connect-flowserver Pod Fails to Mount S3 Secrets

Symptom: The connect-flowserver pod fails to start. To confirm the cause:

  1. Fetch the flow-server pods and identify the failing one:

    kubectl -n foundation-env-default get pods -l app.kubernetes.io/name=flowserver

  2. Describe the failing pod:

    kubectl -n foundation-env-default describe pod <pod-name>

The output shows errors similar to the following:

MountVolume.SetUp failed for volume "secrets": secret "connect-flowserver-s3-creds" not found
MountVolume.SetUp failed for volume "secrets": secret "connect-flowserver-s3-tls" not found

Root Cause: Connect uses global.minio.enabled and global.seaweedfs.enabled to determine which object store credentials to use (see SeaweedFS Connect Migration Guide). If Connect is configured for an object store that Foundation has not deployed and enabled, the source secret (minio1-secret or seaweedfs1-secret) does not exist. So, SecretManager cannot create connect-flowserver-s3-creds or connect-flowserver-s3-tls.

Solution: Verify which object store is enabled in Foundation. Then set the same object store consistently in the connect, connect-postgresql, and connect-openbao Helm values. For required values and validation steps, see SeaweedFS Connect Migration Guide.