listObjectsFromAwsS3

Processor that lists objects from an AWS S3 bucket, for a given object key name prefix.

A resulting list is set as the payload, each element containing a set of metadata. In case of no results, the payload will be set with an empty list.

The following table describes the element structure of a list item:

field name          | type         | description
--------------------|--------------|-----------------
key                 | String       | object key name
lastModified        | Long         | time in epoch millis
eTag                | String       | etag
size                | Long         | size in bytes of the object/blob
storageClass        | String       | name of the enum defined by AWS
checksumType        | String       | name of the enum defined by AWS (only available if S3 provides info)
checksumAlgorithm   | List<String> | name of the enum defined by AWS (only available if S3 provides info)
ownerId             | String       | object's owner id (only available if S3 provides info)
ownerDisplayName    | String       | object's owner display name (only available if S3 provides info)
isRestoreInProgress | Boolean      | flag that indicates whether restore is in progress (only available if S3 provides info)
restoreExpiryDate   | Long         | time in epoch millis (only available if S3 provides info)

Example json compliant payload result:

[
  {
    "key": "foobar",
    "lastModified": 1741867652277,
    "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"",
    "size": 14,
    "storageClass": "STANDARD"
  },
  {
    "key": "foobaz",
    "lastModified": 1741867652304,
    "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"",
    "size": 14,
    "storageClass": "STANDARD"
  },
  {
    "key": "fooquux",
    "lastModified": 1741867652321,
    "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"",
    "size": 14,
    "storageClass": "STANDARD"
  },
  {
    "key": "fooqux",
    "lastModified": 1741867652314,
    "eTag": "\"6d61ed9d869934b23fcad30d800df80f\"",
    "size": 14,
    "storageClass": "STANDARD"
  }
]

The size of the returned list can be configured in listSize. This property has a bounded values set of [1, 1000].

When a list of objects exceeds listSize, pagination is enabled using a continuationToken. If the returned list is truncated, the processor will update this property with the continuationToken for the next page. Before sending a message to the processor, set the s3.list.nextPageToken exchange property with the desired continuationToken to fetch a specific page.

NB: We do not provide list ordering guaranties, that is determined by the S3 implementation.

For AWS S3, the following documentation links state the ordering details:

An example of a set of objects naming convention that utilizes lexicographical ordering would be:

  • "some-object-prefix-20250127-151930.000.json",

  • "some-object-prefix-20250127-151930.001.json",

  • "some-object-prefix-20250127-151930.100.json",

  • "some-object-prefix-20250127-151930.200.json",

  • "some-object-prefix-20250127-151931.000.json",

  • "some-object-prefix-20250127-161930.000.json",

  • "some-object-prefix-20250128-151930.000.json",

  • "some-object-prefix-20250227-151930.000.json",

  • "some-object-prefix-20251027-151930.000.json",

  • "some-object-prefix-20251127-151930.000.json",

  • "some-object-prefix-20251227-151930.000.json",

  • "some-object-prefix-20260227-151930.000.json",

Where the following prefix filtered listings can be executed:

  • To get all files from year 2025, would need a prefix set to some-object-prefix-2025

  • To get all files from January 2025, would need a prefix set to some-object-prefix-202501

  • To get all files from January 27th of 2025, would need a prefix set to some-object-prefix-20250127

Properties

Name Summary

bucketName

The storage bucket name the objects will be listed from. Required.

objectKeyNamePrefixExpression

Optional expression used to specify the object key name prefix for listing objects from the storage bucket

An empty string ( "") prefix is treated as if no prefix is being specified, hence no prefix filtering is applied in listing of objects.

For example, if there is a bucket containing the following objects:

  • Europe/France/Nouvelle-Aquitaine/Bordeaux

  • North America/Canada/Quebec/Montreal

  • North America/USA/Washington/Bellevue

  • North America/USA/Washington/Seattle

To list all the US cities, you should use the following prefixes: 'North America/USA' or 'North America/U'

If this field is not set, the exchange payload will be converted to a string and used as an object key name prefix.

For extra information about object key name prefixes please consult the AWS documentation page: Organizing objects using prefixes

listSize

Optional max number of objects to return per bucket listing command. When it is not configured, the resulting list will have a size that is imposed by the AWS S3 sdk.

The response might contain fewer keys but will never contain more.

This property has a lower bound limit of 1 and an upper bound limit of 1000.

If there are more than 1000 objects the listing can be continued using continuationToken

When the result set is truncated (not complete), a continuationToken is provided in exchange property s3.list.nextPageToken.

The same exchange property s3.list.nextPageToken can be used with a new message to fetch the next results page.

authenticationConfigKey

A secret key that the server uses to look up the credentials needed to perform the AWS authentication. Required.

endpointUrl

Override base URL of the AWS S3 endpoint. Optional. The base URL is automatically detected when interacting with AWS S3, so it’s not necessary to configure it in most of the cases. However, it could be relevant in the following situations:

  1. When using AWS PrivateLink to access S3 within Virtual Private Cloud (VPC) without traversing the public internet, a specific endpoint might be used for that PrivateLink connection.

  2. For applications that require Federal Information Processing Standards (FIPS) compliance, AWS provides FIPS endpoints, which use FIPS 140-2 validated cryptographic modules. These endpoints are different from the standard regional endpoints.

  3. When using S3 in AWS Local Zones or AWS Outposts, specific endpoints associated with those local deployments might be configured.

It is required when running against S3 API services other than AWS, for example MinIO.

Important: Endpoint URL must not specify bucket name as path parameter. Instead, the required bucketName config field must be used. Virtual-hosted-style URLs (i.e. domain containing bucket reference) are not supported. Instead, path-style URLs are used, that require the bucketName field configured.

region

Region name of the bucket. Optional.

retryMaxAttempts

Optional, maximum number of retry attempts the S3 client can perform before failing the request. This option only affects the producer and not the regular flow redelivery mechanism. Defaults to 0, meaning the processor will not perform any retries (though the flow source will still perform retries if configured).

retryInitialDelayMillis

The delay before performing the first retry after the original request has failed. Optional.

retryMaxDelayMillis

Optional, maximum time to wait between retries. An exponential mechanism is used to calculate the next delay between retries.

connectionTimeoutMillis

The timeout of the HTTP client socket connection. Optional.

receiveTimeoutMillis

The socket timeout to wait for the first byte of response from the server. Optional.

sslAuthenticationConfigKey

Key used to look up secret with SSL credentials for server and client authentication. Parameter allowCertificateAssociatedWithWrongHost is ignored on this processor. If using custom trust store, you must provide endpointUrl. Optional.

name

Optional, descriptive name for the processor.

id

Required identifier of the processor, unique across all processors within the flow. Must be between 3 and 30 characters long; contain only lower and uppercase alphabetical characters (a-z and A-Z), numbers, dashes ("-"), and underscores ("_"); and start with an alphabetical character. In other words, it adheres to the regex pattern [a-zA-Z][a-zA-Z0-9_-]{2,29}.

exchangeProperties

Optional set of custom properties in a simple jdk-format, that are added to the message exchange properties before processing the incoming payload. Any existing properties with the same name will be replaced by properties defined here.

retainPayloadOnFailure

Whether the incoming payload is available for error processing on failure. Defaults to false.

Sub-builders

Name Summary

externalSystemDetails

Strategy for describing the external system integration. Optional.

circuitBreakerStrategy

Strategy for configuring the processor’s circuit breaker. Optional.

messageLoggingStrategy

Strategy for describing how a processor’s message is logged on the server.

payloadArchivingStrategy

Strategy for archiving payloads.

inboundTransformationStrategy

Strategy that customizes the conversion of an incoming payload by a processor (e.g., string to object). Should be used when the processor’s default conversion logic cannot be used.

Details

Authentication

The authenticationConfigKey property supports secrets of type AwsCredentials. See the Secret Types documentation for formatting details.