Connect Agent Setup

The GridOS Connect Agent is only supported on version 1.16.0 or later. Make sure you have upgraded your GridOS Connect deployment to a supported version.

System Requirements

Hardware

Connect Agent can only be installed on a 64-bit version of Linux. This can be on a dedicated server or virtual machine but must meet the following criteria:

Hardware Recommended minimum requirements

Processor

Dual 64-bit processors. More processors allow for increased, simultaneous flow executions.

Memory

4 GB of RAM. More RAM allows for increased, simultaneous flow executions and might be required to process large data payloads.

Hard disk

10GB. SSD based storage is recommended for the best performance.

Software

You will need to install Docker (with Docker Compose), version 20.10.0 or higher.

  • Remove unused container images after an upgrade to free up storage:

    docker image prune

  • Also be sure to remove unwanted log archive files. These are stored as .gz files in the logging directory: ${AGENT_INSTALL_DIR}/data/connect-agent/logs.

Create an ID

To get started with Connect Agent, first create an agent ID in Heartbeat:

  1. Select Agents from the left-hand navigation menu: The list of Connect Agent instances includes a name and an auto-generated ID.

  2. Select the Create button in the top right corner and enter a name, which generates the agent ID.

This ID is required for your Connect Agent configuration and client mTLS certificate later in the Agent installation process.

Create Oauth Client Credentials Grant Zitadel Service User

Ensure you have created a Zitadel Service User, ensuring the following:

Set Up the Connect Agent

GE Vernova Artifactory Access

Contact your GE Vernova Support representative for more information on how to get access to GE Vernova Artifactory.

Generate an Artifactory token:

  1. Go to: https://dig-grid-artifactory.apps.ge.com/ui/user_profile

  2. Once logged in, click on Generate an Identity Token. The description should help you remember its purpose, e.g: GridOS Connect Token.

  3. Click Next. The generated token is used as your password credential, with your SSO as the username.

docker login dig-grid-artifactory.apps.ge.com

Deployment Artifacts for Air-Gapped Machines

If you are deploying to a machine with network access to the Artifactory, you can skip this section.

  1. Visit the connect-releases repository.

  2. Download the connect-agent-cache-container-{version}.tgz file to your deployment machine.

  3. Load the cache container contents to your local environment:

    # Load the Connect Agent cache image
docker load < connect-agent-cache-container-{version}.tgz
# Fetch the Connect agent docker images from the cache
docker run --rm -it \
  -v ${PWD}:/app/artifacts \
  --entrypoint bash \
  grid-oslo-docker/connect-agent-cache:{version} -c 'cp data/connect-agent-images.tar.gz artifacts'
# Extract connect-agent-images.tar
tar xvfz connect-agent-images.tar.gz
# Load the 4 Connect Agent images (vector, postgres, connect-agent and connect-agent-starter)
docker load < connect-agent-images.tar
# you can remove the intermediary container images and tarballs to free up storage
docker rmi grid-oslo-docker/connect-agent-cache:{version}
rm connect-agent-cache-container-*.tgz
rm connect-agent-images.tar*

Target Machine Deployment Preparations

  1. On you deployment target Linux machine, create an installation directory. E.g:

    export AGENT_INSTALL_DIR=~/connect-agent-installation
mkdir -p $\{AGENT_INSTALL_DIR\}/cert

  2. Visit the connect-releases repository.

  3. Download the connect-agent-{version}.zip file to your installation directory.

  4. Extract the zip file to your installation directory:

    cd ${AGENT_INSTALL_DIR}
unzip connect-agent-{version}.zip

  5. Copy the server-ca-cert PEM file to ${AGENT_INSTALL_DIR}/cert/ca.crt.

  6. Copy the client-cert PEM file to ${AGENT_INSTALL_DIR}/cert/client.crt.

  7. Copy the client-private-key PEM file to ${AGENT_INSTALL_DIR}/cert/client.key.

Rename (or copy) the connect-agent-template.env file to its new name connect-agent.env.

Each new Connect Agent release will come with a (possible modified) version of this file. Hence, renaming it will ensure that the changes applied to this file will not be overwritten by a new release.

Open the new connect-agent.env file and fill in the following properties:

Property Description

CONNECT_AGENT_ID

The unique ID for this Connect Agent. This ID is provided when adding a new Connect Agent in the Connect Console.

CONNECT_OWNER_ID

The ownerId for the organisation. As used in the deployment directory as well as the deployment config.

CONNECT_DOMAIN

The FQDN for the main GridOS Connect cluster. See YOUR_CONNECT_DOMAIN in our installation documentation. This information can be found in the GridOS Connect Console, under the Agent settings. An image of this is show in the Public Server CA Root Certificate section above.

DEPLOYMENT_ENVIRONMENT

The environment name of the main GridOS Connect cluster. As used in the deployment directory as well as the :deployer.adoc#required-parameters[deployement config].

DECRYPTION_KEY

Decryption key for credentials stored in connect-agent-override.properties. Optional. Only needed if the properties if the the properties contain sensitive information. See how encrypted values are generated for more information.

POSTGRES_PASSWORD

Select a cryptographically secure password to be used for the Connect Agent PostgreSQL server running on the Agent environment. Do not change after first installation.

DEPLOYMENT_PUBLISHER_CLIENT_ID

The client ID of the service user used for publishing to the upstream server API. Same as used for SDK deployer. This pertains to the section above.

DEPLOYMENT_PUBLISHER_CLIENT_SECRET

The client secret of the service user used for publishing to the upstream server API. Same as used for SDK deployer. This pertains to the section above.

DEPLOYMENT_PUBLISHER_SCOPE

The scope used for publishing to the upstream server API. Same as used for SDK deployer. This pertains to the section above.

DEPLOYMENT_PUBLISHER_HTTP_PROXY_URI

URI for HTTP proxy. (Optional)

DEPLOYMENT_PUBLISHER_HTTP_PROXY_USERNAME

The username which will be used for HTTP proxy authentication. (Optional)

DEPLOYMENT_PUBLISHER_HTTP_PROXY_PASSWORD

The password which will be used for HTTP proxy authentication. (Optional)

POLL_FREQUENCY_SECONDS

Number of seconds between each time the Connect Agent polls for new deployments (default is 120 seconds).

Start the Connect Agent

To start the Connect Agent, run the command:

./connect-agent.sh

To verify that the Connect Agent is running, confirm that the connect-agent.sh script completed without errors, and then run:

docker ps

Verify that postgres and connect-agent are healthy and vector is Up.

The connect-agent.sh script passes arguments directly to docker compose. Refer to the examples provided below.
The start-connect-agent.sh script is included in this installation for backward compatibility with older versions. When run, it displays a message directing the user to use connect-agent.sh instead. The start-connect-agent.sh script is automatically removed after the first run of connect-agent.sh.

Stop the Connect Agent

You can stop the Connect Agent using the down argument.

./connect-agent.sh down

Connect Agent Logs

You can view the Connect Agent logs using the logs -f arguments.

./connect-agent.sh logs -f

Monitor the Connect Agent

You can monitor the Connect Agent using one of the following commands:

./connect-agent.sh top
./connect-agent.sh exec vector vector top

Manage the Connect Agent

The Connect Agent flows and resources are deployed differently than in a standard Connect installation. See the Connect Agent Deployments guide for more information.

After you deploy a flow to Connect Agent, its flow traces will appear in Heartbeat alongside your other flows. See the Filtering Connect Agent data for more information. Deployed flows will also appear in the Flows page in Heartbeat.

Manage Service Account

The connect-agent-override file defines service accounts for the Agent instance. Note that all secrets must be BCrypt-encoded. NOTE: The name of a service account is user-defined but must be unique for each account.

Basic Authentication

To authenticate using basic authentication, the username is the service account name, and the password is the plain-text password corresponding to the BCrypt-encoded password.

mc.flow-server.security.users.<Service-Account>.credentials=<bcryptEncodedPassword>

Flow Authentication with API Keys

The service account name is not used for authentication with API keys, but the service account still needs access to the flow. The API key is sent in the x-api-key header.

mc.flow-server.security.users.<Service-Account>.apiKey=<bcryptEncodedApiKey>

BCrypt Encoding of Passwords and API Keys

Use only ASCII characters in the passphrase. Other characters will not be handled correctly. To generate a BCrypt-encoded password or API key, use the following command: 
docker run --rm httpd:alpine \
  sh -c 'htpasswd -nbBC 10 "" "<passphrase>"' | sed 's/\$2y\$/\$2a\$/' | tr -d ':\n'

Replace 10 with the desired cost factor (we recommend 10 or 12) and <passphrase> with the password or API key you want to encode.

Flow Access Control

Flow access control can be configured in the same file; however, it is recommended to use the flow-access.properties file, which can be deployed with the flows. See the Flow Access Properties documentation for more details.

Example configuration:

mc.flow-server.security.users.<Service-Account>.resourceAccesses=my-flow, my-other-flow
mc.flow-server.security.users.<Service-Account>.resourceAccesses=.*

Upgrade the Connect Agent

The Connect Agent installation may be upgraded by following the below steps:

  1. Visit the connect-releases repository.

  2. Optional: Download the connect-agent-cache-container-{version}.tgz file to your deployment machine.

  3. Optional: Load the cache container contents to your local environment (see here for more details).

  4. Download the connect-agent-{version}.zip file to your installation directory.

  5. Extract the zip file to your installation directory (see here for more details).

  6. Start the Connect Agent (see here for more details).

If you are deploying to a machine with network access to the Artifactory, you can skip the optional cache container steps described above.