Payload Types

Processors in an integration flow send and receive data as either strings, binary, or JSON compliant objects. In most cases, you don’t need to be concerned with which processors use which payload types. Data is often automatically converted to the most suitable type as it passes from one processor to the next. It’s important to be aware of the different types, though, in cases where you are manually transforming a payload or a processor can potentially work with more than one type.

JSON Compliant

The default format for payloads is an instance of a Java Map<String, *> or List<Map<String, *>>, where the field types are compatible with JSON standards. This means properties are limited to strings, numbers, booleans, lists/arrays, and other JSON compliant objects. By following these rules, payloads can be treated as regular JSON, making it easier to map data and convert from/to other JSON compliant objects.

When appropriate, JSON compliant objects are automatically generated as the output of a processor and/or as the internal data format to execute the processor. For instance, a dbStatement processor will produce a JSON compliant object or array and can be instructed to interpret an incoming payload as JSON.

XML Compliant

In the flow-server, XML data can also be represented in a JSON compliant format. However, XML compliant JSON objects must follow certain formatting rules to properly capture the XML declaration, namespaces, attributes, text, etc. These rules not only ensure that JSON compliant objects can be converted to XML but provide consistency when processing incoming XML data.

For example, a soapApi processor for SOAP endpoints might receive the following XML input:

<SayHi xmlns="http://www.bccs.uib.no/EchoService.wsdl" xmlns:aa="http://www.bccs-aa.uib.no/EchoService.wsdl">
    <Hi>hi, it's john</Hi>
    <Hi from="jane">hello</Hi>
    <aa:Hi>
        <aa:Greeting>well, hi</aa:Greeting>
        <aa:Speaker>janice</aa:Speaker>
    </aa:Hi>
    <Message xmlns="http://www.bccs-bb.uib.no/EchoService.wsdl">
        <![CDATA[
            Dear Jim,

            What's up?
        ]]>
    </Message>
</SayHi>

The data would then be converted to the following JSON compliant object for the next processor to use:

{
  "_declaration": {
    "version": "1.0",
    "standalone": "no"
  },
  "ns1_SayHi": {
    "ns1_Hi": [
      {
        "_text": "hi, it's john"
      },
      {
        "_attributes": {
          "from": "jane"
        },
        "_text": "hello"
      }
    ],
    "ns2_Hi": [
      {
        "ns2_Greeting": {
          "_text": "well, hi"
        },
        "ns2_Speaker": {
          "_text": "janice"
        }
      }
    ],
    "ns3_Message": {
      "_cdata": "\n    Dear Jim,\n\n    What's up?\n    "
    }
  },
  "_xmlns": {
    "ns1": "http://www.bccs.uib.no/EchoService.wsdl",
    "ns2": "http://www.bccs-aa.uib.no/EchoService.wsdl",
    "ns3": "http://www.bccs-bb.uib.no/EchoService.wsdl"
  }
}

In this example, three XML namespaces were used that are reflected in the _xmlns property and assigned incremental ns prefixes. The <SayHi> and <Hi> elements fall under the first namespace and are therefore accessed as ns1_SayHi and ns1_Hi in the object.

Even though only one <aa:Hi> element is present, the ns2_Hi property is still an array, because the XML schema specifying the format for these elements defined a maxOccurs of more than one. If there isn’t a schema for the converter to consult, it will make the following assumptions about the data:

  • Multiple adjacent elements with the same name are an array.

  • A single element with no like siblings is a regular object.

This is why it’s very important to use an XML schema to avoid any ambiguity.

Any textual content of an element is available on a child _text property. The exception is CDATA, which is placed under a _cdata property. Elements that include attributes are given an additional _attributes property. The following table more easily highlights this conversion between XML and JSON:

XML JSON 
<Hi from="jane">hello</Hi>
 
"ns1_Hi": [
  {
    "_attributes": {
      "from": "jane"
    },
    "_text": "hello"
  }
]

Namespace Mapping

The auto-generated ns prefixes can be overridden by using a namespacePrefixMapping on the processor. The following configuration overrides the prefix for the first two namespaces:

soapApi {
    id = "soap-echo-api"
    wsdlSpecId = soapResourceKey.toResourceIdentifier()
    serviceName = "EchoService"
    portName = "EchoService"
    namespacePrefixMapping = """
        _default = http://www.bccs.uib.no/EchoService.wsdl
        aa = http://www.bccs-aa.uib.no/EchoService.wsdl
    """.trimIndent()
}

Setting a _default removes the prefix altogether, so the converted JSON compliant object now looks like the following:

{
  "_declaration": {
    "version": "1.0",
    "standalone": "no"
  },
  "SayHi": {
    "Hi": [
      {
        "_text": "hi, it's john"
      },
      {
        "_attributes": {
          "from": "jane"
        },
        "_text": "hello"
      }
    ],
    "aa_Hi": [
      {
        "aa_Greeting": {
          "_text": "well, hi"
        },
        "aa_Speaker": {
          "_text": "janice"
        }
      }
    ],
    "ns1_Message": {
      "_cdata": "\n    Dear Jim,\n\n    What's up?\n    "
    }
  },
  "_xmlns": {
    "_default": "http://www.bccs.uib.no/EchoService.wsdl",
    "aa": "http://www.bccs-aa.uib.no/EchoService.wsdl",
    "ns1": "http://www.bccs-bb.uib.no/EchoService.wsdl"
  }
}

Outbound XML

Data intended for a soapRequest processor or as the response in a soapApi flow must follow the same XML compliant JSON format. A simplified version of that format looks like the following:

{
  "_xmlns": {
    "_default": "namespace_1",
    "other": "namespace_2"
  },
  "Element_Name": { // namespace_1
    "Child_Element": {
      "_text": "value"
    }
  },
  "other_Element": { // namespace_2
    "_text": "value"
  }
}

The processor will then be able to interpret and convert the data to valid XML.

Strings

Some processors produce a string instead of a JSON compliant object. A restRequest processor, for example, makes an external request to a REST API and returns the response as a string. Even if the API is JSON-based, the payload still becomes a stringified JSON object.

If restRequest is the last processor in the flow, the client will interpret the string correctly. However, if you need to further process the data, keep in mind that not every processor will automatically convert the string to a JSON compliant object. Processors like map and dbStatement can work just fine with strings, so an automatic conversion isn’t necessarily desired. In such cases, you can manually convert the incoming payload with an inboundTransformationStrategy sub-builder.

Binary

Some processors and flow sources produce binary data. A readFiles inbound endpoint, for example, returns a text file’s contents as a binary byte string.

In most cases, the fact that the data is being processed as binary isn’t important. A restRequest processor will still understand and convert the data to a JSON compliant object. Other processors, like dbStatement, would need a hint first in the form of an inboundTransformationStrategy sub-builder.

Binary Serialization

Binary data may be received or expected to be sent out in a binary serialization format, i.e. Avro. Payload conversion support conversion between Json Compliant and Binary Serialized payloads.

See Payload Conversions section below for the supported serialization formats and usage.

Payload Conversions

To convert an incoming payload to a different format, add an inboundTransformationStrategy sub-builder to the processor. The strategy converts in one of two ways.

  • Conversion by format names the format the payload arrives in and the format the processor receives, such as fromJsonText().toJsonCompliant(). Use it for new flows.

  • Conversion by properties is the older way, and remains for backwards compatibility. It sets properties such as characterSet and objectConversionFormat. The conversion it performs depends on which properties are set and on the type of payload that arrives.

The two ways cannot be mixed. A strategy that mixes them fails when the flow spec is built.

Conversion by Format

A from… call names the format the payload arrives in, and a to… call names the format the processor receives:

inboundTransformationStrategy {
    fromJsonText().toJsonCompliant()
}

Each from… call offers only the to… calls it can convert to. Each call’s sub-builder takes only the options of its own format. The IDE therefore lists every conversion, and the compiler rejects a misplaced option. A strategy holds one conversion.

The following table shows each conversion by format, and the conversion by properties that does the same where one exists:

Conversion Conversion by properties, then by format

Binary to string

 
characterSet = "UTF-8"
fromBytes().toStringPayload()

String to binary

 
characterSet = "ISO-8859-1"
fromStringPayload().toBytes {
    textCharacterSet = "ISO-8859-1"
}

JSON string or binary to object

 
objectConversionFormat = MarshallingFormat.JSON
characterSet = "CESU-8"
fromJsonText {
    textCharacterSet = "CESU-8"
}.toJsonCompliant()

Object to JSON string

 
characterSet = "UTF-8"
fromJsonCompliant().toJsonText()

Object to formatted JSON string

 
characterSet = "UTF-8"
transformerProperties = "formattedOutput=true"
fromJsonCompliant().toJsonText {
    formattedOutput = true
}

List to NDJSON string

 
characterSet = "UTF-8"
transformerProperties = "ndJsonOutput=true"
fromJsonCompliant().toJsonText {
    ndJsonOutput = true
}

XML string or binary to object

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "ISO-8859-1"
fromXmlText {
    textCharacterSet = "ISO-8859-1"
}.toJsonCompliant()

XML with a schema and namespace prefixes to object

 
objectConversionFormat = MarshallingFormat.XML
schemaId = "XSD:echo-service:1"
xmlNamespacePrefixMapping = """
    _default = http://www.bccs.uib.no/EchoService.wsdl
    aa = http://www.bccs-aa.uib.no/EchoService.wsdl
""".trimIndent()
fromXmlText {
    schemaId = "XSD:echo-service:1"
    namespacePrefix("_default", "http://www.bccs.uib.no/EchoService.wsdl")
    namespacePrefix("aa", "http://www.bccs-aa.uib.no/EchoService.wsdl")
}.toJsonCompliant()

XML with insecure processing to object

 
objectConversionFormat = MarshallingFormat.XML
allowInsecureXmlProcessing = true
fromXmlText {
    allowInsecureProcessing = true
}.toJsonCompliant()

XML object to XML string

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "UTF-8"
fromJsonCompliant().toXmlText()

XML object to XML binary, on a processor that sends it

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "ISO-8859-1"
fromJsonCompliant().toXmlText {
    textCharacterSet = "ISO-8859-1"
}

Object to Avro binary (single record)

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"
fromJsonCompliant().toAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}

Object (list) to Avro binary (multiple records)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"
fromJsonCompliant().toAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}

Avro binary (single record) to object

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"
fromAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

Avro binary (multiple records) to object (list)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"
fromAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

HDB archive binary to object

Conversion by properties has none.

fromHdbArchive {
    database("BLS") {
        table("AREA")
        table("EVR") { fields("ID_*") }
    }
}.toJsonCompliant()

HDB Archive Conversion describes the selection and the object.

Each option is optional, except schemaId on an Avro conversion:

Option Taken by Description

textCharacterSet

fromBytes, toBytes, fromJsonText, toJsonText, fromXmlText, toXmlText

The character set of text held as bytes. Defaults to UTF-8.

formattedOutput

toJsonText

If true, the JSON text is indented. Defaults to false.

ndJsonOutput

toJsonText

If true, a list is written as newline-delimited JSON, one item to a line, and any payload that is not a list fails. Defaults to false.

schemaId

fromXmlText

The IDs of the XSD or WSDL resources that describe the XML, separated by commas, such as XSD:echo-service:1. The object then follows the schema. For example, it holds a list for an element that may occur more than once.

schemaId

fromAvroRecords, fromAvroSingleRecord, toAvroRecords, toAvroSingleRecord

Required. The ID of one Avro Schema resource, such as AvroSchema:mySchema:latest. Each Avro item must conform to it.

namespacePrefix(prefix, namespace)

fromXmlText

Gives a namespace its prefix in the object, as Namespace Mapping describes. The prefix _default names the namespace whose elements get no prefix. Each call maps one prefix, and a namespace takes one prefix.

database(name) { table(name) { fields(patterns) } }

fromHdbArchive

The selection: the databases, tables and fields to convert, as HDB Archive Conversion describes. Leaving it out converts the whole archive.

allowInsecureProcessing

fromXmlText

If true, allows insecure XML features, such as doctype and entity resolution. Defaults to false. It takes effect only where the flow-server’s allowInsecureXmlProcessing feature toggle is on. Every message the processor converts fails where the toggle is off.

textCharacterSet decodes the bytes that fromBytes, fromJsonText and fromXmlText read, and encodes the bytes that toBytes writes. Under toJsonText and toXmlText, the processor always receives a string. A processor that sends that string as bytes, such as sendToKafka, encodes it in textCharacterSet. toXmlText sets the encoding of the object’s XML declaration to textCharacterSet.

A payload that is already an object under toJsonCompliant, a string under toStringPayload or binary under toBytes passes through unchanged. Every other payload must be in the from format, or the message fails with an error that names both formats and the payload’s type. A string fails under toJsonText and toXmlText, because those conversions read only an object. Avro arrives as binary only, so a string fails under fromAvroRecords and fromAvroSingleRecord. An HDB archive arrives as binary only too, so a string fails under fromHdbArchive.

toAvroRecords and toAvroSingleRecord give the processor binary. fromAvroRecords and toAvroRecords read and write a list, with one Avro item to each entry. fromAvroSingleRecord and toAvroSingleRecord read and write a single object. Avro Conversion describes both.

A conversion by properties with an object format and a character set converts both ways. It picks the direction from the payload: binary becomes an object, and an object becomes a string. A conversion by format converts one way only. To replace such a strategy, use fromJsonText() or fromXmlText() when the processor gets binary or a string, and fromJsonCompliant() when it gets an object.

A conversion by properties that turns an object into a JSON string in a character set other than UTF-8 garbles every non-ASCII character. A conversion by properties that writes an XML string loses each character the set cannot hold. Its XML declaration copies encoding from the object’s _declaration, even where the text is encoded in another character set. toJsonText and toXmlText keep every character, in any character set.

Conversion by format was introduced in Connect 1.33.0, and cannot be used with Connect deployments on earlier versions. Their flow-servers ignore the from and to fields, and run the processor without a conversion.

HDB Archive Conversion

An HDB archive is a file in which Habitat, a GE Vernova SCADA application, saves a copy of its database. fromHdbArchive { }.toJsonCompliant() turns an archive into an object that holds its data. The fromHdbArchive sub-builder holds the selection: the databases, tables and fields to convert. Leave the sub-builder out, as in fromHdbArchive().toJsonCompliant(), to convert every table. Conversion by properties cannot read an HDB archive, and no conversion writes one.

inboundTransformationStrategy {
    fromHdbArchive {
        database("BLS") {
            table("AREA")
            table("EVR") { fields("ID_*") }
        }
        database("BLSLOG") { table("MESS") }
    }.toJsonCompliant()
}

In the selection:

  • database(name) converts that database. Without a table sub-builder, it converts all the database’s tables.

  • table(name) converts that table. Without a fields call, it converts all the table’s fields.

  • fields(patterns) converts the fields that match the patterns. * matches any characters, so ID_* matches every field that starts with ID_.

Write database and table names in full. Only field patterns take *. Names are case-sensitive, and Habitat writes them in upper case. A pseudo-table, such as RL%, is converted only when a table names it. A * in a database or table name, or a name given twice in one sub-builder, fails when the flow spec is built or read from JSON.

The selection picks columns, not rows: each table comes with all its rows. Each row also keeps the columns that place it in the table tree, which the list below describes. To keep only some rows, add a processor after the conversion that filters them.

A name or pattern that matches nothing in the archive fails the message. The error lists what the archive holds. For example, table("ITEM") in an archive whose database ALARMDEF holds the table ITEMS fails with:

the payload holds no table matching ITEM in the databases matching ALARMDEF; those hold [ITEMS, RL%, RH%, RP%]

The object holds databases, with one entry for each database converted:

{
  "databases": [
    {
      "name": "LOGMAN",
      "tables": [
        {
          "name": "QUEUE",
          "fields": ["subscript", "parentSubscript", "MAX_QUEUE", "NAME_QUEUE"],
          "data": [[1, 1, 3000, "EVENTQ"], [2, 1, 20, "FILEQ"]],
          "fieldsWithoutData": []
        }
      ],
      "tableTree": [
        { "table": "DECK", "parentTable": null, "level": 2 },
        { "table": "QUEUE", "parentTable": "DECK", "level": 3 }
      ]
    }
  ]
}

  • tables holds the converted tables. Without a selection, that is every table with data, except the pseudo-tables Habitat keeps for itself. fields names a table’s columns once. data holds one list of values for each row, in the order fields names them.

  • Habitat arranges its tables in a tree, and a row of one table can belong to a row of the table above it. The first columns of each row give its place in that tree. subscript is the row’s number in its table, counting from one. parentSubscript is the number of its parent row, where the table hangs below another.

  • tableTree lists every table in the database, pseudo-tables included, with the table each hangs below. It always lists them all, whatever the selection names.

  • A null in data is either a value the database does not hold, or a float that is NaN or infinite. The two look the same. fieldsWithoutData names the fields the archive holds no data for.

Two things differ from what Habitat’s own tools print:

  • An array field comes back as a list indexed from zero. Habitat’s tools number the same elements from one.

  • A timestamp comes back in UTC, as an ISO-8601 instant such as 2002-06-26T19:25:14Z. Habitat’s tools print it in the time zone they are set to.

The archive must arrive as binary in memory, not as a stream. A stream that reaches a processor with an inboundTransformationStrategy fails the message. So a flow that reads archives with readFiles leaves streamingEnabled off. A payload that is not an archive, or a damaged one, fails the message with an error that says what is wrong.

Conversion by Properties

Conversion by properties is the older way to configure the strategy, and remains for backwards compatibility. It sets the conversion through properties:

inboundTransformationStrategy {
    objectConversionFormat = <marshalling format>
    characterSet = <character encoding>
}
Refer to Oracle’s Internationalization Guide for a list of supported characterSet encodings.

The following example demonstrates the transformation sub-builder on a map processor, where the incoming payload is expected to be a stringified object with a results property:

map {
    id = "format-response"
    mapSpec = """
        {
            "message" : #input.payload.results[[1]]
        }
    """.trimIndent()
    inboundTransformationStrategy {
        objectConversionFormat = MarshallingFormat.JSON
    }
}

The following table highlights the inboundTransformationStrategy properties needed to perform other possible payload conversions:

From To inboundTransformationStrategy

String

Object

 
objectConversionFormat = MarshallingFormat.JSON

String

XML Object

 
objectConversionFormat = MarshallingFormat.XML

Binary

Object

 
objectConversionFormat = MarshallingFormat.JSON
characterSet = "CESU-8" // default: UTF-8

Binary

String

 
characterSet = "UTF-8"

Object

String

 
characterSet = "UTF-8"

XML Object

XML String

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "UTF-8"

Object

Avro binary (single record)

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"

Object (list)

Avro binary (multiple records)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"

Avro binary (single record)

Object

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"

Avro binary (multiple records)

Object (list)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"

Note: schemaId points to an identifier in Resource Registry.

If you use an inboundTransformationStrategy on an incoming payload that doesn’t fit the desired conversion format, an exception will be thrown.

Avro Conversion

Avro conversion relies on two MarshallingFormats:

  • AVRO_RECORDS: Use if your Avro payload consist of multiple items, e.g. multiple cloud events. The Avro binary will be converted to a list of objects with one entry for each of the Avro items. When converting to Avro the payload must be a list. Each of the list entries will be converted to an Avro item. The conversion will fail if the payload is not a list.

  • AVRO_SINGLE_RECORD: Use if your Avro payload consists of a single item, e.g. a single cloud event. The Avro item will be converted to a single object. The conversion will fail if the Avro binary contains more than one item. When converting to Avro the payload will be converted to a single item.

The Avro transformation does not currently support the proprietary Confluent wire format.

Each Avro item must conform to the provided Avro Schema.

Transformation Properties

String transformations that occur from an object can be further configured using the transformerProperties property. For example:

inboundTransformationStrategy {
    characterSet = "UTF-8"
    transformerProperties = """
        propertyOne=true
        propertyTwo=true
    """.trimIndent()
}

The following properties are available:

Transformer Property Marshalling Format Description

formattedOutput

JSON, JAXB, XML

If true, the marshalled data is formatted with linefeed and indentation. Defaults to false.

jaxbWindowsLineEnding

JAXB

If true, replaces UNIX line breaks (\n) with DOS/Windows line breaks (\r\n). Defaults to false.

ndJsonOutput

JSON

If true, the output is formatted as NDJSON (i.e., newline delimited JSON). Input must be a JSON compliant array of objects. Defaults to false.

xmlWriteDatesAsTimestamps

XML

If true, datetime values are serialized as numeric timestamps (epoch timestamps). Defaults to false (textual representation).