listObjectsFromAwsS3
Processor that lists objects from an AWS S3 bucket, for a given object key name prefix.
A resulting list is set as the payload, each element containing a set of metadata. In case of no results, the payload will be set with an empty list.
The following table describes the element structure of a list item:
field name | type | description --------------------|--------------|----------------- key | String | object key name lastModified | Long | time in epoch millis eTag | String | etag size | Long | size in bytes of the object/blob storageClass | String | name of the enum defined by AWS checksumType | String | name of the enum defined by AWS (only available if S3 provides info) checksumAlgorithm | List<String> | name of the enum defined by AWS (only available if S3 provides info) ownerId | String | object's owner id (only available if S3 provides info) ownerDisplayName | String | object's owner display name (only available if S3 provides info) isRestoreInProgress | Boolean | flag that indicates whether restore is in progress (only available if S3 provides info) restoreExpiryDate | Long | time in epoch millis (only available if S3 provides info)
Example json compliant payload result:
[ { "key": "foobar", "lastModified": 1741867652277, "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"", "size": 14, "storageClass": "STANDARD" }, { "key": "foobaz", "lastModified": 1741867652304, "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"", "size": 14, "storageClass": "STANDARD" }, { "key": "fooquux", "lastModified": 1741867652321, "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"", "size": 14, "storageClass": "STANDARD" }, { "key": "fooqux", "lastModified": 1741867652314, "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"", "size": 14, "storageClass": "STANDARD" } ]
The size of the returned list can be configured in listSize
. This property has a bounded values set of [1, 1000]
.
When a list of objects exceeds listSize
, pagination is enabled using a continuationToken. If the returned list is truncated, the processor will update this property with the continuationToken for the next page. Before sending a message to the processor, set the s3.list.nextPageToken
exchange property with the desired continuationToken to fetch a specific page.
NB: We do not provide list ordering guaranties, that is determined by the S3 implementation.
For AWS S3, the following documentation links state the ordering details:
An example of a set of objects naming convention that utilizes lexicographical ordering would be:
-
"some-object-prefix-20250127-151930.000.json",
-
"some-object-prefix-20250127-151930.001.json",
-
"some-object-prefix-20250127-151930.100.json",
-
"some-object-prefix-20250127-151930.200.json",
-
"some-object-prefix-20250127-151931.000.json",
-
"some-object-prefix-20250127-161930.000.json",
-
"some-object-prefix-20250128-151930.000.json",
-
"some-object-prefix-20250227-151930.000.json",
-
"some-object-prefix-20251027-151930.000.json",
-
"some-object-prefix-20251127-151930.000.json",
-
"some-object-prefix-20251227-151930.000.json",
-
"some-object-prefix-20260227-151930.000.json",
Where the following prefix filtered listings can be executed:
-
To get all files from year 2025, would need a prefix set to
some-object-prefix-2025
-
To get all files from January 2025, would need a prefix set to
some-object-prefix-202501
-
To get all files from January 27th of 2025, would need a prefix set to
some-object-prefix-20250127
Properties
Name | Summary |
---|---|
|
The storage bucket name the objects will be listed from. Required. |
|
Optional expression used to specify the object key name prefix for listing objects from the storage bucket An empty string ( For example, if there is a bucket containing the following objects:
To list all the US cities, you should use the following prefixes: 'North America/USA' or 'North America/U' If this field is not set, the exchange payload will be converted to a string and used as an object key name prefix. For extra information about object key name prefixes please consult the AWS documentation page: Organizing objects using prefixes |
|
Optional max number of objects to return per bucket listing command. When it is not configured, the resulting list will have a size that is imposed by the AWS S3 sdk. The response might contain fewer keys but will never contain more. This property has a lower bound limit of 1 and an upper bound limit of If there are more than When the result set is truncated (not complete), a The same exchange property |
|
A secret key that the server uses to look up the credentials needed to perform the AWS authentication. Required. |
|
Base URL of the AWS S3 endpoint. Optional. |
|
Region name of the bucket. Optional. |
|
Optional, maximum number of retry attempts the S3 client can perform before failing the request. This option only affects the producer and not the regular flow redelivery mechanism. Defaults to |
|
The delay before performing the first retry after the original request has failed. Optional. |
|
Optional, maximum time to wait between retries. An exponential mechanism is used to calculate the next delay between retries. |
|
The timeout of the HTTP client socket connection. Optional. |
|
The socket timeout to wait for the first byte of response from the server. Optional. |
|
Key used to look up secret with SSL credentials for server and client authentication. Parameter allowCertificateAssociatedWithWrongHost is ignored on this processor. If using custom trust store, you must provide endpointUrl. Optional. |
|
Optional, descriptive name for the processor. |
|
Required identifier of the processor, unique across all processors within the flow. Must be between 3 and 30 characters long; contain only lower and uppercase alphabetical characters (a-z and A-Z), numbers, dashes ("-"), and underscores ("_"); and start with an alphabetical character. In other words, it adheres to the regex pattern |
|
Optional set of custom properties in a simple jdk-format, that are added to the message exchange properties before processing the incoming payload. Any existing properties with the same name will be replaced by properties defined here. |
|
Whether the incoming payload is available for error processing on failure. Defaults to |
Sub-builders
Name | Summary |
---|---|
Strategy for describing the external system integration. Optional. |
|
Strategy for configuring the processor’s circuit breaker. Optional. |
|
Strategy for describing how a processor’s message is logged on the server. |
|
Strategy for archiving payloads. |
|
Strategy that customizes the conversion of an incoming payload by a processor (e.g., string to object). Should be used when the processor’s default conversion logic cannot be used. |
Details
Authentication
The authenticationConfigKey
property supports secrets of type AwsCredentials.
See the Secret Types documentation for formatting details.