Payload Types

Processors in an integration flow send and receive data as either strings, binary, or JSON compliant objects. In most cases, you don’t need to be concerned with which processors use which payload types. Data is often automatically converted to the most suitable type as it passes from one processor to the next. It’s important to be aware of the different types, though, in cases where you are manually transforming a payload or a processor can potentially work with more than one type.

JSON Compliant

The default format for payloads is an instance of a Java Map<String, *> or List<Map<String, *>>, where the field types are compatible with JSON standards. This means properties are limited to strings, numbers, booleans, lists/arrays, and other JSON compliant objects. By following these rules, payloads can be treated as regular JSON, making it easier to map data and convert from/to other JSON compliant objects.

When appropriate, JSON compliant objects are automatically generated as the output of a processor and/or as the internal data format to execute the processor. For instance, a dbStatement processor will produce a JSON compliant object or array and can be instructed to interpret an incoming payload as JSON.

XML Compliant

In the flow-server, XML data can also be represented in a JSON compliant format. However, XML compliant JSON objects must follow certain formatting rules to properly capture the XML declaration, namespaces, attributes, text, etc. These rules not only ensure that JSON compliant objects can be converted to XML but provide consistency when processing incoming XML data.

For example, a soapApi processor for SOAP endpoints might receive the following XML input:

<SayHi xmlns="http://www.bccs.uib.no/EchoService.wsdl" xmlns:aa="http://www.bccs-aa.uib.no/EchoService.wsdl">
    <Hi>hi, it's john</Hi>
    <Hi from="jane">hello</Hi>
    <aa:Hi>
        <aa:Greeting>well, hi</aa:Greeting>
        <aa:Speaker>janice</aa:Speaker>
    </aa:Hi>
    <Message xmlns="http://www.bccs-bb.uib.no/EchoService.wsdl">
        <![CDATA[
            Dear Jim,

            What's up?
        ]]>
    </Message>
</SayHi>

The data would then be converted to the following JSON compliant object for the next processor to use:

{
  "_declaration": {
    "version": "1.0",
    "standalone": "no"
  },
  "ns1_SayHi": {
    "ns1_Hi": [
      {
        "_text": "hi, it's john"
      },
      {
        "_attributes": {
          "from": "jane"
        },
        "_text": "hello"
      }
    ],
    "ns2_Hi": [
      {
        "ns2_Greeting": {
          "_text": "well, hi"
        },
        "ns2_Speaker": {
          "_text": "janice"
        }
      }
    ],
    "ns3_Message": {
      "_cdata": "\n    Dear Jim,\n\n    What's up?\n    "
    }
  },
  "_xmlns": {
    "ns1": "http://www.bccs.uib.no/EchoService.wsdl",
    "ns2": "http://www.bccs-aa.uib.no/EchoService.wsdl",
    "ns3": "http://www.bccs-bb.uib.no/EchoService.wsdl"
  }
}

In this example, three XML namespaces were used that are reflected in the _xmlns property and assigned incremental ns prefixes. The <SayHi> and <Hi> elements fall under the first namespace and are therefore accessed as ns1_SayHi and ns1_Hi in the object.

Even though only one <aa:Hi> element is present, the ns2_Hi property is still an array, because the XML schema specifying the format for these elements defined a maxOccurs of more than one. If there isn’t a schema for the converter to consult, it will make the following assumptions about the data:

  • Multiple adjacent elements with the same name are an array.

  • A single element with no like siblings is a regular object.

This is why it’s very important to use an XML schema to avoid any ambiguity.

Any textual content of an element is available on a child _text property. The exception is CDATA, which is placed under a _cdata property. Elements that include attributes are given an additional _attributes property. The following table more easily highlights this conversion between XML and JSON:

XML JSON 
<Hi from="jane">hello</Hi>
 
"ns1_Hi": [
  {
    "_attributes": {
      "from": "jane"
    },
    "_text": "hello"
  }
]

Namespace Mapping

The auto-generated ns prefixes can be overridden by using a namespacePrefixMapping on the processor. The following configuration overrides the prefix for the first two namespaces:

soapApi {
    id = "soap-echo-api"
    wsdlSpecId = soapResourceKey.toResourceIdentifier()
    serviceName = "EchoService"
    portName = "EchoService"
    namespacePrefixMapping = """
        _default = http://www.bccs.uib.no/EchoService.wsdl
        aa = http://www.bccs-aa.uib.no/EchoService.wsdl
    """.trimIndent()
}

Setting a _default removes the prefix altogether, so the converted JSON compliant object now looks like the following:

{
  "_declaration": {
    "version": "1.0",
    "standalone": "no"
  },
  "SayHi": {
    "Hi": [
      {
        "_text": "hi, it's john"
      },
      {
        "_attributes": {
          "from": "jane"
        },
        "_text": "hello"
      }
    ],
    "aa_Hi": [
      {
        "aa_Greeting": {
          "_text": "well, hi"
        },
        "aa_Speaker": {
          "_text": "janice"
        }
      }
    ],
    "ns1_Message": {
      "_cdata": "\n    Dear Jim,\n\n    What's up?\n    "
    }
  },
  "_xmlns": {
    "_default": "http://www.bccs.uib.no/EchoService.wsdl",
    "aa": "http://www.bccs-aa.uib.no/EchoService.wsdl",
    "ns1": "http://www.bccs-bb.uib.no/EchoService.wsdl"
  }
}

Outbound XML

Data intended for a soapRequest processor or as the response in a soapApi flow must follow the same XML compliant JSON format. A simplified version of that format looks like the following:

{
  "_xmlns": {
    "_default": "namespace_1",
    "other": "namespace_2"
  },
  "Element_Name": { // namespace_1
    "Child_Element": {
      "_text": "value"
    }
  },
  "other_Element": { // namespace_2
    "_text": "value"
  }
}

The processor will then be able to interpret and convert the data to valid XML.

Strings

Some processors produce a string instead of a JSON compliant object. A restRequest processor, for example, makes an external request to a REST API and returns the response as a string. Even if the API is JSON-based, the payload still becomes a stringified JSON object.

If restRequest is the last processor in the flow, the client will interpret the string correctly. However, if you need to further process the data, keep in mind that not every processor will automatically convert the string to a JSON compliant object. Processors like map and dbStatement can work just fine with strings, so an automatic conversion isn’t necessarily desired. In such cases, you can manually convert the incoming payload with an inboundTransformationStrategy sub-builder.

Binary

Some processors and flow sources produce binary data. A readFiles inbound endpoint, for example, returns a text file’s contents as a binary byte string.

In most cases, the fact that the data is being processed as binary isn’t important. A restRequest processor will still understand and convert the data to a JSON compliant object. Other processors, like dbStatement, would need a hint first in the form of an inboundTransformationStrategy sub-builder.

Binary serialization

Binary data may be received or expected to be sent out in a binary serialization format, i.e. Avro. Payload conversion support conversion between Json Compliant and Binary Serialized payloads.

See Payload Conversions section below for the supported serialization formats and usage.

Payload Conversions

In cases where you need an incoming payload converted to a different format, you can add the following inboundTransformationStrategy sub-builder to the relevant processor:

inboundTransformationStrategy {
    objectConversionFormat = <marshalling format>
    characterSet = <character encoding>
}
Refer to Oracle’s Internationalization Guide for a list of supported characterSet encodings.

The following example demonstrates the transformation sub-builder on a map processor, where the incoming payload is expected to be a stringified object with a results property:

map {
    id = "format-response"
    mapSpec = """
        {
            "message" : #input.payload.results[[1]]
        }
    """.trimIndent()
    inboundTransformationStrategy {
        objectConversionFormat = MarshallingFormat.JSON
    }
}

The following table highlights the inboundTransformationStrategy properties needed to perform other possible payload conversions:

From To inboundTransformationStrategy

String

Object

 
objectConversionFormat = MarshallingFormat.JSON

String

XML Object

 
objectConversionFormat = MarshallingFormat.XML

Binary

Object

 
objectConversionFormat = MarshallingFormat.JSON
characterSet = "CESU-8" // default: UTF-8

Binary

String

 
characterSet = "UTF-8"

Object

String

 
characterSet = "UTF-8"

XML Object

XML String

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "UTF-8"

Object

Avro binary (single record)

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"

Object (list)

Avro binary (multiple records)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"

Avro binary (single record)

Object

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"

Avro binary (multiple records)

Object (list)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"

Note: schemaId points to an identifier in Resource Registry.

If you use an inboundTransformationStrategy on an incoming payload that doesn’t fit the desired conversion format, an exception will be thrown.

Avro conversion

Avro conversion relies on two MarshallingFormats:

  • AVRO_RECORDS: Use if your Avro payload consist of multiple items, e.g. multiple cloud events. The Avro binary will be converted to a list of objects with one entry for each of the Avro items. When converting to Avro the payload must be a list. Each of the list entries will be converted to an Avro item. The conversion will fail if the payload is not a list.

  • AVRO_SINGLE_RECORD: Use if your Avro payload consists of a single item, e.g. a single cloud event. The Avro item will be converted to a single object. The conversion will fail if the Avro binary contains more than one item. When converting to Avro the payload will be converted to a single item.

Each Avro item must conform to the provided Avro Schema.

Transformation Properties

String transformations that occur from an object can be further configured using the transformerProperties property. For example:

inboundTransformationStrategy {
    characterSet = "UTF-8"
    transformerProperties = """
        propertyOne=true
        propertyTwo=true
    """.trimIndent()
}

The following properties are available:

Transformer Property Marshalling Format Description

formattedOutput

JSON, JAXB, XML

If true, the marshalled data is formatted with linefeed and indentation. Defaults to false.

jaxbWindowsLineEnding

JAXB

If true, replaces UNIX line breaks (\n) with DOS/Windows line breaks (\r\n). Defaults to false.

ndJsonOutput

JSON

If true, the output is formatted as NDJSON (i.e., newline delimited JSON). Input must be a JSON compliant array of objects. Defaults to false.

xmlWriteDatesAsTimestamps

XML

If true, datetime values are serialized as numeric timestamps (epoch timestamps). Defaults to false (textual representation).