sendToKafka

Required list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This string must be in the form of host1:port1,host2:port2, etc.

Name of the topic to which the message should be published. Required.

Number of acknowledgments the producer requires the leader to have received before considering a request complete. Available options are defined on the KafkaAcks enum. Optional and defaults to the value defined in Kafka ( PARTIAL).

The upper bound on the time to report success or failure after a call to send() returns. According to Kafka docs, this value should be greater than lingerPeriodMs plus requestTimeoutMs. In other words: deliveryTimeoutMs>= lingerPeriodMs + requestTimeoutMs. Optional and defaults to the value from the flow server config.

The upper bound on the time to get metadata from the broker. Used to verify connection to the server, as well as obtain information about the cluster. This value should be kept considerably low, since it controls how fast a connection attempt will fail when the backend has problems. Optional and defaults to the value from the flow server config.

The maximum amount of time (in milliseconds) the client will wait for the response of a request. Optional and defaults to the value from the flow server config.

Specifies how long the producer will wait for messages before sending them to the broker. It is used to batch several messages together in a single request, reducing traffic to the broker. Messages will be sent once either the lingerPeriodMs time has passed or the batchSize limit is reached. Optional and defaults to the value from the flow server config.

Upper bound on the batch size in bytes. If there is a linger period, the producer will try to put several messages in a single batch until a batchSize limit is reached. Once either the lingerPeriodMs time has passed or the batchSize limit is reached, the request will be sent to the broker. Optional and defaults to the value from the flow server config.

The KafkaCompressionMode for all data generated by the producer. Recommended value is ZSTD. Optional and defaults to the value from the flow server config.

Optional, maximum number of unacknowledged requests the client will send on a single connection before blocking. The client usually creates one connection per broker in the cluster.

The recommended strategy for this parameter is to use no less than half of the potential requests per broker in the cluster. This can be determined with the following equations:

For example, when the Kafka producer receives 2000 messages of about 600 bytes each, has a batch size of 16000 bytes, and must deliver it to a cluster of 12 brokers, the following math applies:

When the Kafka producer receives 2 messages of about 8000000 bytes each, has a batch size of 16000 bytes, and must deliver it to a cluster of 3 brokers, the following math applies:

When the Kafka producer receives 2000 messages of about 600 bytes each, has a batch size of 16000 bytes, and must deliver it to a cluster of 1 broker, the following math applies:

Key used to look up the credentials needed to connect to the Kafka broker. Optional and uses no authentication by default.

Optional, descriptive name for the processor.

Required identifier of the processor, unique across all processors within the flow. Must be between 3 and 30 characters long; contain only lower and uppercase alphabetical characters (a-z and A-Z), numbers, dashes ("-"), and underscores ("_"); and start with an alphabetical character. In other words, it adheres to the regex pattern [a-zA-Z][a-zA-Z0-9_-]{2,29}.

Optional set of custom properties in a simple jdk-format, that are added to the message exchange properties before processing the incoming payload. Any existing properties with the same name will be replaced by properties defined here.

Whether the incoming payload is available for error processing on failure. Defaults to false.

Strategy for providing message processing hints to the server. Optional.

Strategy for describing the external system integration. Optional.

Strategy for configuring the processor’s circuit breaker. Optional.

Strategy for describing how a processor’s message is logged on the server.

Strategy for archiving payloads.

Strategy that customizes the conversion of an incoming payload by a processor (e.g., string to object). Should be used when the processor’s default conversion logic cannot be used.