Payload Conversions

To convert an incoming payload to a different format, add an inboundTransformationStrategy sub-builder to the processor. Use Conversion by Format for new integration flows. Payload Types explains the payload representations used below.

Existing property-based strategies are documented in Conversion by Properties (Deprecated).

For legacy string-formatting options, see Transformation Properties.

You can’t mix conversion by format and conversion by properties in the same strategy. A strategy that mixes the two forms fails when the flow spec is built.

Conversion by Format (New in 2.0.0)

A from… call names the format the payload arrives in, and a to… call names the format the processor receives:

inboundTransformationStrategy {
    fromJsonText().toJsonCompliant()
}

Conversion by format supports these formats:

Format Calls Payload

JSON-compliant payload

fromJsonCompliant, toJsonCompliant

An in-memory JSON-compliant object or list.

JSON text

fromJsonText, toJsonText

JSON, as a string or binary.

XML text

fromXmlText, toXmlText

XML, as a string or binary. Converted to an object, it follows the XML compliant rules.

Avro, multiple records

fromAvroRecords, toAvroRecords

Avro binary with one record for each entry of a list. See Avro Conversion.

Avro, single record

fromAvroSingleRecord, toAvroSingleRecord

Avro binary with exactly one record, which can be an object or a whole list.

HDB archive

fromHdbArchive

A copy of Habitat’s database, called HDB. The conversion can read an archive but not write one, so there’s no to… call. See HDB Archive Conversion.

Plain text

fromPlainText, toPlainText

A string, with no format of its own.

Binary

fromBytes, toBytes

Bytes, with no format of their own.

A strategy holds one conversion. Each from… call offers only the to… calls that it can convert to. Each call’s sub-builder, the block in braces after it, accepts only the options for its own format. Your IDE suggests only the conversions and options that fit, and the compiler rejects the rest.

The following examples show every conversion. Each commented example is a separate strategy. Only Avro conversions require schemaId; all other options are optional. fromJsonCompliant and toJsonCompliant take no options.

toJsonCompliant passes through a JSON-compliant object or list unchanged. toPlainText does the same for a string, and toBytes for binary. Any other payload must be in the from format. Otherwise, the message fails with an error that names both formats and the payload type.

Connect 2.0.0 introduced conversion by format. Flow-servers on earlier versions ignore the from and to fields and run the processor without a conversion.

JSON Conversion

fromJsonText { }.toJsonCompliant() converts JSON text, held as a string or binary, to a JSON-compliant object or list. The fromJsonText sub-builder accepts:

Option Description

characterSet

The character set the conversion uses to decode JSON text held as bytes. Choose one from Oracle’s list of supported encodings. Defaults to UTF-8.

// JSON text, as a string or binary, to a JSON-compliant payload
fromJsonText { characterSet = "UTF-16" }.toJsonCompliant()

fromJsonCompliant().toJsonText { } converts a JSON-compliant object or list to JSON text. The toJsonText sub-builder accepts:

Option Description

characterSet

The character set a processor such as sendToKafka uses to send the JSON text as bytes. Defaults to UTF-8.

formattedOutput

If true, the conversion indents the JSON text. Defaults to false.

ndJsonOutput

If true, writes a list as newline-delimited JSON, with one item on each line. A payload that is not a list fails. Defaults to false.

The processor receives JSON text as a string, even when you set characterSet. A string arriving under toJsonText fails because the conversion reads only a JSON-compliant object or list.

// JSON-compliant payload to JSON text
fromJsonCompliant().toJsonText()

// JSON-compliant payload to indented JSON text
fromJsonCompliant().toJsonText { formattedOutput = true }

// List to newline-delimited JSON (NDJSON) text
fromJsonCompliant().toJsonText { ndJsonOutput = true }

XML Conversion

fromXmlText { }.toJsonCompliant() converts XML text, held as a string or binary, to a JSON-compliant object that follows the XML-compliant formatting rules. The fromXmlText sub-builder accepts:

Option Description

characterSet

The character set the conversion uses to decode XML text held as bytes. Defaults to UTF-8.

schemaId

Optional. The IDs of the XSD or WSDL resources that describe the XML, separated by commas, such as XSD:echo-service:1. The object then follows the schema. For example, an element that can occur more than once becomes a list.

namespacePrefix(prefix, namespace)

Sets the prefix for a namespace in the object. See Namespace Mapping. The prefix _default names the namespace whose elements get no prefix. Each call maps one prefix, and a namespace can have only one prefix.

allowInsecureProcessing

If true, allows insecure XML features, such as doctype and entity resolution. Defaults to false. Without it, XML that carries a doctype declaration fails the message. Set it only for XML from a source you trust, because entity resolution exposes the flow-server to XML external entity (XXE) attacks. An operator must also set mc.flow-server.featureToggles.allowInsecureXmlProcessing to true in the flow-server configuration. If you enable allowInsecureProcessing without that toggle, every message that the processor converts fails.

// XML text, as a string or binary, to a JSON-compliant object
fromXmlText { characterSet = "ISO-8859-1" }.toJsonCompliant()

// XML with a schema and namespace prefixes to a JSON-compliant object
fromXmlText {
    schemaId = "XSD:echo-service:1"
    // "_default" names the namespace whose elements get no prefix
    namespacePrefix("_default", "http://www.bccs.uib.no/EchoService.wsdl")
    namespacePrefix("aa", "http://www.bccs-aa.uib.no/EchoService.wsdl")
}.toJsonCompliant()

// XML with a doctype declaration to a JSON-compliant object
fromXmlText { allowInsecureProcessing = true }.toJsonCompliant()

fromJsonCompliant().toXmlText { } converts an XML-compliant object to XML text. The toXmlText sub-builder accepts:

Option Description

characterSet

The character set a processor such as sendToKafka uses to send the XML text as bytes. Defaults to UTF-8. Also sets the encoding of the object’s XML declaration.

The processor receives XML text as a string, even when you set characterSet. A string arriving under toXmlText fails because the conversion reads only an XML-compliant object.

// XML-compliant object to XML text
fromJsonCompliant().toXmlText()

// XML-compliant object to XML text, for a processor that sends it as ISO-8859-1 bytes
fromJsonCompliant().toXmlText { characterSet = "ISO-8859-1" }

Avro Conversion

Avro conversion reads binary into a JSON-compliant payload or writes a JSON-compliant payload as binary. A string fails under fromAvroRecords and fromAvroSingleRecord. The two pairs of builders differ in how many records the Avro binary holds.

Multiple Records

toAvroRecords writes one record for each entry of a list, and fails on a payload that is not a list. fromAvroRecords reads every record into a list, with one entry for each. Both sub-builders require:

Option Description

schemaId

Required. The ID of one Avro Schema resource, such as AvroSchema:mySchema:latest. Each Avro record must conform to it.

Use these builders when the binary holds several records, such as several cloud events. Under toAvroRecords, the processor receives binary.

// Avro binary (multiple records) to a JSON-compliant list
fromAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

// JSON-compliant list to Avro binary (multiple records)
fromJsonCompliant().toAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}

Single Record

toAvroSingleRecord writes the whole JSON-compliant object or list as one record. fromAvroSingleRecord reads one record into a JSON-compliant object or list. Binary with no records or more than one record fails the message. Both sub-builders require:

Option Description

schemaId

Required. The ID of one Avro Schema resource, such as AvroSchema:mySchema:latest. The Avro record must conform to it.

Use these builders when the binary holds one record, such as one cloud event. Under toAvroSingleRecord, a list payload therefore needs a schema that describes a list. The processor receives binary.

// Avro binary (single record) to a JSON-compliant object or list
fromAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

// JSON-compliant object or list to Avro binary (single record)
fromJsonCompliant().toAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}
Avro conversion does not support the proprietary Confluent wire format.

HDB Archive Conversion

An HDB archive is a file in which Habitat, a GE Vernova SCADA application, saves a copy of its database (HDB). fromHdbArchive { }.toJsonCompliant() converts an archive into an object that contains its data. The fromHdbArchive sub-builder selects the databases, tables, and fields to convert. It accepts:

Option Description

database(name) { table(name) { fields(patterns) } }

Selects the databases, tables, and fields to convert. If you omit the selection, the conversion includes the whole archive.

To convert the whole archive, omit the sub-builder, as in fromHdbArchive().toJsonCompliant(). Conversion by format can read an HDB archive but can’t write one. An archive must arrive as binary, so a string fails under fromHdbArchive.

inboundTransformationStrategy {
    fromHdbArchive {
        database("BLS") {
            table("AREA")
            table("LEVEL") { fields("ID_*") }
        }
        database("BLSLOG") { table("MESS") }
    }.toJsonCompliant()
}

In the selection:

  • database(name) converts that database. Without a table sub-builder, it converts all the database’s tables.

  • table(name) converts that table. Without a fields call, it converts all the table’s fields.

  • fields(patterns) converts the fields that match the patterns. * matches any characters, so ID_* matches every field that starts with ID_.

Write database and table names in full. Only field patterns accept *. Names are case-sensitive, and Habitat writes them in upper case. The conversion includes a pseudo-table, such as RL%, only if a table call names it. The flow spec fails to build, or to read from JSON, if a database or table name contains * or if a name appears twice in one sub-builder.

The selection filters columns, not rows. Each table includes all its rows. Each row also keeps the columns that place it in the table tree, described in the list below. To keep only some rows, add a processor after the conversion to filter them.

A name or pattern that matches nothing in the archive fails the message. The error lists what the archive holds. For example, table("ITEM") in an archive whose database ALARMDEF holds the table ITEMS fails with:

the payload holds no table matching ITEM in the databases matching ALARMDEF; those hold [ITEMS, RL%, RH%, RP%]

The resulting object contains databases, with one entry for each converted database:

{
  "databases": [
    {
      "name": "LOGMAN",
      "tables": [
        {
          "name": "QUEUE",
          "fields": ["subscript", "parentSubscript", "MAX_QUEUE", "NAME_QUEUE"],
          "data": [[1, 1, 3000, "EVENTQ"], [2, 1, 20, "FILEQ"]],
          "fieldsWithoutData": []
        }
      ],
      "tableTree": [
        { "table": "DECK", "parentTable": null, "level": 2 },
        { "table": "QUEUE", "parentTable": "DECK", "level": 3 }
      ]
    }
  ]
}
  • tables contains the converted tables. Without a selection, that is every table with data, except the pseudo-tables that Habitat keeps for itself. fields names a table’s columns once. data contains one list of values for each row, in the order that fields lists them.

  • Habitat arranges its tables in a tree, and a row of one table can belong to a row of the table above it. The first columns of each row give its place in that tree. subscript is the row’s number in its table, counting from one. parentSubscript is the number of its parent row, where the table hangs below another.

  • tableTree lists every table in the database, pseudo-tables included, with the table each hangs below. It always lists them all, whatever the selection names.

  • A null in data is either a value the database does not hold, or a float that is NaN or infinite. The two look the same. fieldsWithoutData names the fields the archive holds no data for.

Two things differ from what Habitat’s own tools print:

  • An array field comes back as a list indexed from zero. Habitat’s tools number the same elements from one.

  • A timestamp comes back in UTC, as an ISO-8601 instant such as 2002-06-26T19:25:14Z. Habitat’s tools print it in the time zone they are set to.

An inboundTransformationStrategy does not support streamed payloads, so keep streamingEnabled off in a flow that reads archives with readFiles. A payload that is not an archive, or is damaged, fails the message.

An archive usually contains compressed data, which can grow to many times the archive’s size during decompression. The conversion streams the decompressed archive into a temporary file, then loads the selected data into memory. A large selection can take more memory than the flow-server has.

Select only the data the flow needs. Convert a whole archive only when the flow uses all of its data. Throttle the flow with maxNumberOfInFlightMessages and maxBufferSize so it does not run out of memory. For details, see Throttling.

The conversion writes the temporary file to the Java temporary directory (java.io.tmpdir) unless an operator sets a different directory with mc.flow-server.hdb-archive.temp-directory in the flow-server configuration.

Plain Text Conversion

fromBytes { }.toPlainText() decodes binary to a plain-text string. toPlainText takes no options. The fromBytes sub-builder accepts:

Option Description

characterSet

The character set the conversion uses to decode the bytes. Defaults to UTF-8.

// Binary to plain text
fromBytes().toPlainText()

Binary Conversion

fromPlainText().toBytes { } encodes a plain-text string as binary. fromPlainText takes no options. The toBytes sub-builder accepts:

Option Description

characterSet

The character set the conversion uses to encode the text as bytes. Defaults to UTF-8.

// Plain text to binary
fromPlainText().toBytes { characterSet = "ISO-8859-1" }