Payload Conversions

To convert an incoming payload to a different format, add an inboundTransformationStrategy sub-builder to the processor. You can configure the strategy in one of two ways:

  • Conversion by format names the format the payload arrives in and the format the processor receives, such as fromJsonText().toJsonCompliant(). Use it for new flows.

  • Conversion by properties is deprecated and remains only for backward compatibility. It sets properties such as characterSet and objectConversionFormat. The conversion depends on which properties you set and on the type of the incoming payload.

You can’t mix the two ways. A strategy that mixes them fails when the flow spec is built.

Conversion by Format

A from… call names the format the payload arrives in, and a to… call names the format the processor receives:

inboundTransformationStrategy {
    fromJsonText().toJsonCompliant()
}

Conversion by format supports these formats:

Format Calls Payload

Object

fromJsonCompliant, toJsonCompliant

A JSON compliant object or list.

JSON text

fromJsonText, toJsonText

JSON, as a string or binary.

XML text

fromXmlText, toXmlText

XML, as a string or binary. Converted to an object, it follows the XML compliant rules.

Avro, multiple records

fromAvroRecords, toAvroRecords

Avro binary with one record for each entry of a list. See Avro Conversion.

Avro, single record

fromAvroSingleRecord, toAvroSingleRecord

Avro binary with exactly one record, which can be an object or a whole list.

HDB archive

fromHdbArchive

A copy of Habitat’s database, called HDB. The conversion can read an archive but not write one, so there’s no to… call. See HDB Archive Conversion.

String

fromStringPayload, toStringPayload

A string, with no format of its own.

Binary

fromBytes, toBytes

Bytes, with no format of their own.

A strategy holds one conversion. Each from… call offers only the to… calls that it can convert to. Each call’s sub-builder, the block in braces after it, accepts only the options for its own format. Your IDE suggests only the conversions and options that fit, and the compiler rejects the rest.

The following examples show every conversion. Each commented example is a separate strategy. All options are optional except schemaId on an Avro conversion. The table after the examples describes each option.

To convert a payload to an object:

// JSON string or binary to object
fromJsonText { textCharacterSet = "UTF-16" }.toJsonCompliant()

// XML string or binary to object
fromXmlText { textCharacterSet = "ISO-8859-1" }.toJsonCompliant()

// XML with a schema and namespace prefixes to object
fromXmlText {
    schemaId = "XSD:echo-service:1"
    // "_default" names the namespace whose elements get no prefix
    namespacePrefix("_default", "http://www.bccs.uib.no/EchoService.wsdl")
    namespacePrefix("aa", "http://www.bccs-aa.uib.no/EchoService.wsdl")
}.toJsonCompliant()

// XML with a doctype declaration to object
fromXmlText { allowInsecureProcessing = true }.toJsonCompliant()

// Avro binary (single record) to object or list
fromAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

// Avro binary (multiple records) to list
fromAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

// HDB archive binary to object
fromHdbArchive {
    database("BLS") {
        table("AREA")
        table("EVR") { fields("ID_*") }
    }
}.toJsonCompliant()

HDB Archive Conversion and Avro Conversion describe the selection in the fromHdbArchive sub-builder, and the object it produces.

To convert an object to a string or binary:

// Object to JSON string
fromJsonCompliant().toJsonText()

// Object to formatted JSON string
fromJsonCompliant().toJsonText { formattedOutput = true }

// List to newline-delimited JSON (NDJSON) string
fromJsonCompliant().toJsonText { ndJsonOutput = true }

// XML compliant object to XML string
fromJsonCompliant().toXmlText()

// XML compliant object to XML string, for sending as ISO-8859-1 bytes
fromJsonCompliant().toXmlText { textCharacterSet = "ISO-8859-1" }

// Object or list to Avro binary (single record)
fromJsonCompliant().toAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}

// List to Avro binary (multiple records)
fromJsonCompliant().toAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}

To convert between a string and binary:

// Binary to string
fromBytes().toStringPayload()

// String to binary
fromStringPayload().toBytes { textCharacterSet = "ISO-8859-1" }

The options are:

Option Taken by Description

textCharacterSet

fromBytes, toBytes, fromJsonText, toJsonText, fromXmlText, toXmlText

The character set of text held as bytes, one of those in Oracle’s list of supported encodings. Defaults to UTF-8.

formattedOutput

toJsonText

If true, the JSON text is indented. Defaults to false.

ndJsonOutput

toJsonText

If true, writes a list as newline-delimited JSON, with one item on each line. A payload that is not a list fails. Defaults to false.

schemaId

fromXmlText

Optional. The IDs of the XSD or WSDL resources that describe the XML, separated by commas, such as XSD:echo-service:1. The object then follows the schema. For example, an element that can occur more than once becomes a list.

schemaId

fromAvroRecords, fromAvroSingleRecord, toAvroRecords, toAvroSingleRecord

Required. The ID of one Avro Schema resource, such as AvroSchema:mySchema:latest. Each Avro record must conform to it.

namespacePrefix(prefix, namespace)

fromXmlText

Sets the prefix for a namespace in the object. See Namespace Mapping. The prefix _default names the namespace whose elements get no prefix. Each call maps one prefix, and a namespace can have only one prefix.

database(name) { table(name) { fields(patterns) } }

fromHdbArchive

Selects the databases, tables, and fields to convert. See HDB Archive Conversion. If you omit the selection, the conversion includes the whole archive.

allowInsecureProcessing

fromXmlText

If true, allows insecure XML features, such as doctype and entity resolution. Defaults to false. Without it, XML that carries a doctype declaration fails the message. Set it only for XML from a source you trust, because entity resolution exposes the flow-server to XML external entity (XXE) attacks. An operator must also set mc.flow-server.featureToggles.allowInsecureXmlProcessing to true in the flow-server configuration. Without that toggle, every message that the processor converts fails.

textCharacterSet decodes the bytes that fromBytes, fromJsonText, and fromXmlText read, and encodes the bytes that toBytes writes. Under toJsonText and toXmlText, the processor always receives a string that keeps every character. A processor that sends that string as bytes, such as sendToKafka, encodes it in textCharacterSet. toXmlText sets the encoding of the object’s XML declaration to textCharacterSet.

A payload that is already in the target format passes through unchanged: an object under toJsonCompliant, a string under toStringPayload, or binary under toBytes. Any other payload must be in the from format. Otherwise, the message fails with an error that names both formats and the payload type. A string fails under toJsonText and toXmlText because those conversions read only an object. Avro data arrives only as binary, so a string fails under fromAvroRecords and fromAvroSingleRecord. An HDB archive also arrives only as binary, so a string fails under fromHdbArchive.

Conversion by format was introduced in Connect 1.33.0. Flow-servers on earlier versions ignore the from and to fields and run the processor without a conversion.

Avro Conversion

The Avro calls come in two pairs, which differ in how many records the Avro binary holds:

  • toAvroRecords writes one record for each entry of a list, and fails on a payload that is not a list. fromAvroRecords reads every record into a list, with one entry for each. Use them when the binary holds several records, such as several cloud events.

  • toAvroSingleRecord writes the whole payload as one record, so a list becomes a single record that holds the list. fromAvroSingleRecord reads that one record, which may itself be a list, and fails unless the binary holds exactly one. Use them when the binary holds one record, such as one cloud event.

Each record must conform to the Avro Schema that schemaId names. Under toAvroSingleRecord, a list payload therefore needs a schema that describes a list. Under toAvroRecords and toAvroSingleRecord, the processor receives binary.

Avro conversion does not support the proprietary Confluent wire format.

HDB Archive Conversion

An HDB archive is a file in which Habitat, a GE Vernova SCADA application, saves a copy of its database (HDB). fromHdbArchive { }.toJsonCompliant() converts an archive into an object that contains its data. The fromHdbArchive sub-builder selects the databases, tables, and fields to convert. To convert every table, omit the sub-builder, as in fromHdbArchive().toJsonCompliant(). Conversion by format can read an HDB archive but can’t write one.

inboundTransformationStrategy {
    fromHdbArchive {
        database("BLS") {
            table("AREA")
            table("EVR") { fields("ID_*") }
        }
        database("BLSLOG") { table("MESS") }
    }.toJsonCompliant()
}

In the selection:

  • database(name) converts that database. Without a table sub-builder, it converts all the database’s tables.

  • table(name) converts that table. Without a fields call, it converts all the table’s fields.

  • fields(patterns) converts the fields that match the patterns. * matches any characters, so ID_* matches every field that starts with ID_.

Write database and table names in full. Only field patterns accept *. Names are case-sensitive, and Habitat writes them in upper case. The conversion includes a pseudo-table, such as RL%, only if a table call names it. The flow spec fails to build, or to read from JSON, if a database or table name contains * or if a name appears twice in one sub-builder.

The selection filters columns, not rows. Each table includes all its rows. Each row also keeps the columns that place it in the table tree, described in the list below. To keep only some rows, add a processor after the conversion to filter them.

A name or pattern that matches nothing in the archive fails the message. The error lists what the archive holds. For example, table("ITEM") in an archive whose database ALARMDEF holds the table ITEMS fails with:

the payload holds no table matching ITEM in the databases matching ALARMDEF; those hold [ITEMS, RL%, RH%, RP%]

The resulting object contains databases, with one entry for each converted database:

{
  "databases": [
    {
      "name": "LOGMAN",
      "tables": [
        {
          "name": "QUEUE",
          "fields": ["subscript", "parentSubscript", "MAX_QUEUE", "NAME_QUEUE"],
          "data": [[1, 1, 3000, "EVENTQ"], [2, 1, 20, "FILEQ"]],
          "fieldsWithoutData": []
        }
      ],
      "tableTree": [
        { "table": "DECK", "parentTable": null, "level": 2 },
        { "table": "QUEUE", "parentTable": "DECK", "level": 3 }
      ]
    }
  ]
}

  • tables contains the converted tables. Without a selection, that is every table with data, except the pseudo-tables that Habitat keeps for itself. fields names a table’s columns once. data contains one list of values for each row, in the order that fields lists them.

  • Habitat arranges its tables in a tree, and a row of one table can belong to a row of the table above it. The first columns of each row give its place in that tree. subscript is the row’s number in its table, counting from one. parentSubscript is the number of its parent row, where the table hangs below another.

  • tableTree lists every table in the database, pseudo-tables included, with the table each hangs below. It always lists them all, whatever the selection names.

  • A null in data is either a value the database does not hold, or a float that is NaN or infinite. The two look the same. fieldsWithoutData names the fields the archive holds no data for.

Two things differ from what Habitat’s own tools print:

  • An array field comes back as a list indexed from zero. Habitat’s tools number the same elements from one.

  • A timestamp comes back in UTC, as an ISO-8601 instant such as 2002-06-26T19:25:14Z. Habitat’s tools print it in the time zone they are set to.

An inboundTransformationStrategy does not support streamed payloads, so keep streamingEnabled off in a flow that reads archives with readFiles. A payload that is not an archive, or is damaged, fails the message.

An archive is usually compressed, and its data can grow to many times the archive’s size when decompressed. The conversion streams the decompressed archive into a temporary file, then loads the selected data into memory. A large selection can take more memory than the flow-server has.

Select only the data the flow needs. Convert a whole archive only when the flow uses all of its data. Throttle the flow with maxNumberOfInFlightMessages and maxBufferSize so it does not run out of memory. For details, see Throttling.

The conversion writes the temporary file to the Java temporary directory (java.io.tmpdir) unless an operator sets a different directory with mc.flow-server.hdb-archive.temp-directory in the flow-server configuration.

Conversion by Properties

Conversion by properties is deprecated, and remains for backwards compatibility. Use conversion by format for new flows.

Conversion by properties sets the conversion through properties:

inboundTransformationStrategy {
    objectConversionFormat = <marshalling format>
    characterSet = <character encoding>
}
Refer to Oracle’s Internationalization Guide for a list of supported characterSet encodings.

The following example demonstrates the transformation sub-builder on a map processor, where the incoming payload is expected to be a stringified object with a results property:

map {
    id = "format-response"
    mapSpec = """
        {
            "message" : #input.payload.results[[1]]
        }
    """.trimIndent()
    inboundTransformationStrategy {
        objectConversionFormat = MarshallingFormat.JSON
    }
}

A conversion by properties with an objectConversionFormat and a characterSet converts both ways. It picks the direction from the payload: binary becomes an object, and an object becomes a string. Conversion by format has no single equivalent, because it converts one way only. Replace such a strategy with fromJsonText() or fromXmlText() where the payload arrives as binary or a string. Use fromJsonCompliant() where it arrives as an object.

The following table shows the properties for each conversion, and the conversion by format that replaces them:

From To Conversion by properties Conversion by format

String

Object

 
objectConversionFormat = MarshallingFormat.JSON
 
fromJsonText().toJsonCompliant()

String

XML compliant object

 
objectConversionFormat = MarshallingFormat.XML
 
fromXmlText().toJsonCompliant()

Binary XML

XML compliant object

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "ISO-8859-1"
 
fromXmlText {
    textCharacterSet = "ISO-8859-1"
}.toJsonCompliant()

String or binary XML, with a schema and namespace prefixes

XML compliant object

 
objectConversionFormat = MarshallingFormat.XML
schemaId = "XSD:echo-service:1"
xmlNamespacePrefixMapping = """
    _default = http://www.bccs.uib.no/EchoService.wsdl
    aa = http://www.bccs-aa.uib.no/EchoService.wsdl
""".trimIndent()
 
fromXmlText {
    schemaId = "XSD:echo-service:1"
    namespacePrefix("_default", "http://www.bccs.uib.no/EchoService.wsdl")
    namespacePrefix("aa", "http://www.bccs-aa.uib.no/EchoService.wsdl")
}.toJsonCompliant()

String or binary XML, with a doctype declaration

XML compliant object

 
objectConversionFormat = MarshallingFormat.XML
allowInsecureXmlProcessing = true
 
fromXmlText {
    allowInsecureProcessing = true
}.toJsonCompliant()

Binary

Object

 
objectConversionFormat = MarshallingFormat.JSON
characterSet = "UTF-16" // default: UTF-8
 
fromJsonText {
    textCharacterSet = "UTF-16"
}.toJsonCompliant()

Binary

String

 
characterSet = "UTF-8"
 
fromBytes().toStringPayload()

String

Binary

 
characterSet = "ISO-8859-1"
 
fromStringPayload().toBytes {
    textCharacterSet = "ISO-8859-1"
}

Object

String

 
characterSet = "UTF-8"
 
fromJsonCompliant().toJsonText()

XML compliant object

XML String

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "UTF-8"
 
fromJsonCompliant().toXmlText()

XML compliant object

XML String, for sending as ISO-8859-1 bytes

 
objectConversionFormat = MarshallingFormat.XML
characterSet = "ISO-8859-1"
 
fromJsonCompliant().toXmlText {
    textCharacterSet = "ISO-8859-1"
}

Object or list

Avro binary (single record)

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"
 
fromJsonCompliant().toAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}

List

Avro binary (multiple records)

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"
 
fromJsonCompliant().toAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}

Avro binary (single record)

Object or list

 
objectConversionFormat = MarshallingFormat.AVRO_SINGLE_RECORD
schemaId = "AvroSchema:mySchema:latest"
 
fromAvroSingleRecord {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()

Avro binary (multiple records)

List

 
objectConversionFormat = MarshallingFormat.AVRO_RECORDS
schemaId = "AvroSchema:mySchema:latest"
 
fromAvroRecords {
    schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()
schemaId points to an identifier in Resource Registry.
If you use an inboundTransformationStrategy on an incoming payload that doesn’t fit the desired conversion format, an exception will be thrown.

A conversion by properties writes an object’s XML in characterSet, but always writes its JSON in UTF-8. A processor that receives the text as a string gets those bytes decoded in characterSet. Under any character set other than UTF-8, a JSON string therefore garbles every non-ASCII character. An XML string loses each character the set cannot hold.

The XML declaration that a conversion by properties writes takes its encoding from the object’s _declaration property, which XML Compliant describes. The declaration can therefore name a character set other than the one the text is encoded in.

Under conversion by format, toJsonText and toXmlText give the processor a string that keeps every character, and toXmlText sets the declaration’s encoding to textCharacterSet.

MarshallingFormat.AVRO_RECORDS converts like fromAvroRecords and toAvroRecords, and MarshallingFormat.AVRO_SINGLE_RECORD like fromAvroSingleRecord and toAvroSingleRecord. Avro Conversion describes both.

Transformation Properties

String transformations that occur from an object can be further configured using the transformerProperties property. For example:

inboundTransformationStrategy {
    characterSet = "UTF-8"
    transformerProperties = """
        propertyOne=true
        propertyTwo=true
    """.trimIndent()
}

The following properties are available:

Transformer Property Description Conversion by format

formattedOutput

If true, the JSON string is indented. Defaults to false.

fromJsonCompliant().toJsonText { formattedOutput = true }

ndJsonOutput

If true, the output is formatted as NDJSON (i.e., newline delimited JSON). Input must be a JSON compliant array of objects. Defaults to false.

fromJsonCompliant().toJsonText { ndJsonOutput = true }