writeToAwsS3

Processor that writes payloads to an AWS S3 bucket.

The name of the object is set to flowOwner/flowId/messageId by default. This behaviour can be overridden by using the standard fileName message exchange property.

The maximum object size is 5GB!

Properties

Name Summary

Name	Summary
`forwardProperty()`	Registers the named message exchange-property to be written as user-defined metadata on the S3 object produced by this processor. The exchange-property’s name and value are placed verbatim into the object’s user-defined metadata map; the AWS S3 service applies the standard `x-amz-meta-` prefix on the wire automatically, so the name passed here must NOT include that prefix. Passing the wildcard `""` forwards every exchange-property present on the incoming message as metadata; any explicitly named properties are then redundant. An exchange-property listed by name that is not present on the incoming message is silently skipped (no empty metadata entry is written). Calling this function multiple times accumulates the set of property names to forward. Caveats: Property names must be valid HTTP header tokens (ASCII letters, digits and a limited set of separators). Names containing whitespace or non-token characters will cause a request signing or validation failure at write-time. AWS S3 lowercases user-defined metadata keys on the wire; a property forwarded as `FooBar` will be readable only as `foobar` on the resulting object. The total size of user-defined metadata (sum of all key and value bytes) is capped by S3 at 2 KB. Exceeding this limit causes the write to fail with `MetadataTooLarge`, which this processor reports as a flow configuration error. If a forwarded property collides with `contentType` (set either explicitly or from `defaultContentType`), the value will be written both as the standard `Content-Type` header and as a user-defined metadata entry. The wildcard `""` is interpreted only on the write side; the matching read-side processors ( `readFromS3`, `lookupOnS3`, `listObjectsOnS3`) treat `"*"` as a literal property name when used in their own `forwardProperty` configuration. Optional. Defaults to forwarding no properties.
`bucketName`	The storage bucket name under which the payload is stored. Required.
`authenticationConfigKey`	A secret key the server uses to look up the credentials needed to perform the AWS authentication. Required.
`endpointUrl`	Override base URL of the AWS S3 endpoint. Optional. The base URL is automatically detected when interacting with AWS S3, so it’s not necessary to configure it in most of the cases. However, it could be relevant in the following situations: When using AWS PrivateLink to access S3 within Virtual Private Cloud (VPC) without traversing the public internet, a specific endpoint might be used for that PrivateLink connection. For applications that require Federal Information Processing Standards (FIPS) compliance, AWS provides FIPS endpoints, which use FIPS 140-2 validated cryptographic modules. These endpoints are different from the standard regional endpoints. When using S3 in AWS Local Zones or AWS Outposts, specific endpoints associated with those local deployments might be configured. It is required when running against S3 API services other than AWS, for example MinIO. Important: Endpoint URL must not specify bucket name as path parameter. Instead, the required `bucketName` config field must be used. Virtual-hosted-style URLs (i.e. domain containing bucket reference) are not supported. Instead, path-style URLs are used, that require the `bucketName` field configured.
`region`	Region name of the bucket. Optional.
`retryMaxAttempts`	The maximum number of retry attempts the S3 client can perform before failing the request. This option affects only the producer and not the regular flow redelivery mechanism. Optional and defaults to `0`, meaning the processor will not perform any retries (though the flow source will still perform retries if configured).
`retryInitialDelayMillis`	The delay before performing the first retry after the original request has failed. Optional.
`retryMaxDelayMillis`	The maximum time to wait between retries. Optional. An exponential mechanism is used to calculate the next delay between retries.
`retainPayload`	Retain the processor input payload after writing it to the S3 storage, copying the payload to the processor output makes it available for processing in downstream processors. Optional and defaults to `false` (i.e., the payload is removed after it is written to S3).
`defaultContentType`	Default content MIME type (e.g., `application/json; charset=utf-8`) to associate with the payload. Content type can also be defined dynamically by setting the standard `contentType` exchange property. Optional.
`connectionTimeoutMillis`	The timeout of the HTTP client socket connection. Optional.
`receiveTimeoutMillis`	The socket timeout to wait for the first byte of response from the server. Optional.
`sslAuthenticationConfigKey`	A key the server uses to look up secret with SSL credentials for server and client authentication. Parameter allowCertificateAssociatedWithWrongHost is ignored on this processor. If using custom trust store, you must provide endpointUrl. Optional.
`retainPayloadOnFailure`	Whether the incoming payload is available for error processing on failure. Defaults to `false`.
`name`	Optional, descriptive name for the processor.
`id`	Required identifier of the processor, unique across all processors within the flow. Must be between 3 and 30 characters long; contain only lower and uppercase alphabetical characters (a-z and A-Z), numbers, dashes ("-"), and underscores ("_"); and start with an alphabetical character. In other words, it adheres to the regex pattern `[a-zA-Z][a-zA-Z0-9_-]{2,29}`.
`exchangeProperties`	Optional set of custom properties in a simple jdk-format, that are added to the message exchange properties before processing the incoming payload. Any existing properties with the same name will be replaced by properties defined here.

forwardProperty()

Registers the named message exchange-property to be written as user-defined metadata on the S3 object produced by this processor.

The exchange-property’s name and value are placed verbatim into the object’s user-defined metadata map; the AWS S3 service applies the standard x-amz-meta- prefix on the wire automatically, so the name passed here must NOT include that prefix.

Passing the wildcard "*" forwards every exchange-property present on the incoming message as metadata; any explicitly named properties are then redundant. An exchange-property listed by name that is not present on the incoming message is silently skipped (no empty metadata entry is written). Calling this function multiple times accumulates the set of property names to forward.

Caveats:

Property names must be valid HTTP header tokens (ASCII letters, digits and a limited set of separators). Names containing whitespace or non-token characters will cause a request signing or validation failure at write-time.
AWS S3 lowercases user-defined metadata keys on the wire; a property forwarded as FooBar will be readable only as foobar on the resulting object.
The total size of user-defined metadata (sum of all key and value bytes) is capped by S3 at 2 KB. Exceeding this limit causes the write to fail with MetadataTooLarge, which this processor reports as a flow configuration error.
If a forwarded property collides with contentType (set either explicitly or from defaultContentType), the value will be written both as the standard Content-Type header and as a user-defined metadata entry.
The wildcard "*" is interpreted only on the write side; the matching read-side processors ( readFromS3, lookupOnS3, listObjectsOnS3) treat "*" as a literal property name when used in their own forwardProperty configuration.

Optional. Defaults to forwarding no properties.

bucketName

The storage bucket name under which the payload is stored. Required.

authenticationConfigKey

A secret key the server uses to look up the credentials needed to perform the AWS authentication. Required.

endpointUrl

Override base URL of the AWS S3 endpoint. Optional. The base URL is automatically detected when interacting with AWS S3, so it’s not necessary to configure it in most of the cases. However, it could be relevant in the following situations:

When using AWS PrivateLink to access S3 within Virtual Private Cloud (VPC) without traversing the public internet, a specific endpoint might be used for that PrivateLink connection.
For applications that require Federal Information Processing Standards (FIPS) compliance, AWS provides FIPS endpoints, which use FIPS 140-2 validated cryptographic modules. These endpoints are different from the standard regional endpoints.
When using S3 in AWS Local Zones or AWS Outposts, specific endpoints associated with those local deployments might be configured.

It is required when running against S3 API services other than AWS, for example MinIO.

Important: Endpoint URL must not specify bucket name as path parameter. Instead, the required bucketName config field must be used. Virtual-hosted-style URLs (i.e. domain containing bucket reference) are not supported. Instead, path-style URLs are used, that require the bucketName field configured.

region

Region name of the bucket. Optional.

retryMaxAttempts

The maximum number of retry attempts the S3 client can perform before failing the request. This option affects only the producer and not the regular flow redelivery mechanism. Optional and defaults to 0, meaning the processor will not perform any retries (though the flow source will still perform retries if configured).

retryInitialDelayMillis

The delay before performing the first retry after the original request has failed. Optional.

retryMaxDelayMillis

The maximum time to wait between retries. Optional. An exponential mechanism is used to calculate the next delay between retries.

retainPayload

Retain the processor input payload after writing it to the S3 storage, copying the payload to the processor output makes it available for processing in downstream processors. Optional and defaults to false (i.e., the payload is removed after it is written to S3).

defaultContentType

Default content MIME type (e.g., application/json; charset=utf-8) to associate with the payload. Content type can also be defined dynamically by setting the standard contentType exchange property. Optional.

connectionTimeoutMillis

The timeout of the HTTP client socket connection. Optional.

receiveTimeoutMillis

The socket timeout to wait for the first byte of response from the server. Optional.

sslAuthenticationConfigKey

A key the server uses to look up secret with SSL credentials for server and client authentication. Parameter allowCertificateAssociatedWithWrongHost is ignored on this processor. If using custom trust store, you must provide endpointUrl. Optional.

retainPayloadOnFailure

Whether the incoming payload is available for error processing on failure. Defaults to false.

name

Optional, descriptive name for the processor.

id

Required identifier of the processor, unique across all processors within the flow. Must be between 3 and 30 characters long; contain only lower and uppercase alphabetical characters (a-z and A-Z), numbers, dashes ("-"), and underscores ("_"); and start with an alphabetical character. In other words, it adheres to the regex pattern [a-zA-Z][a-zA-Z0-9_-]{2,29}.

exchangeProperties

Optional set of custom properties in a simple jdk-format, that are added to the message exchange properties before processing the incoming payload. Any existing properties with the same name will be replaced by properties defined here.

Sub-builders

Name	Summary
externalSystemDetails	Strategy for describing the external system integration. Optional.
circuitBreakerStrategy	Strategy for configuring the processor’s circuit breaker. Optional.
inboundTransformationStrategy	Strategy that customizes the conversion of an incoming payload by a processor (e.g., string to object). Should be used when the processor’s default conversion logic cannot be used.
messageLoggingStrategy	Strategy for describing how a processor’s message is logged on the server.
payloadArchivingStrategy	Strategy for archiving payloads.

Name

Summary

externalSystemDetails

Strategy for describing the external system integration. Optional.

circuitBreakerStrategy

Strategy for configuring the processor’s circuit breaker. Optional.

inboundTransformationStrategy

Strategy that customizes the conversion of an incoming payload by a processor (e.g., string to object). Should be used when the processor’s default conversion logic cannot be used.

messageLoggingStrategy

Strategy for describing how a processor’s message is logged on the server.

payloadArchivingStrategy

Strategy for archiving payloads.

Details

Authentication

The authenticationConfigKey property supports secrets of type AwsCredentials. See the Secret Types documentation for formatting details.

Was this page helpful?