Payload Conversions
To convert an incoming payload to a different format, add an inboundTransformationStrategy sub-builder to the processor.
Use Conversion by Format for new integration flows.
Payload Types explains the payload representations used below.
Existing property-based strategies are documented in Conversion by Properties (Deprecated).
For legacy string-formatting options, see Transformation Properties.
You can’t mix conversion by format and conversion by properties in the same strategy. A strategy that mixes the two forms fails when the flow spec is built.
Conversion by Format (New in 2.0.0)
A from… call names the format the payload arrives in, and a to… call names the format the processor receives:
inboundTransformationStrategy {
fromJsonText().toJsonCompliant()
}
Conversion by format supports these formats:
| Format | Calls | Payload |
|---|---|---|
JSON-compliant payload |
|
An in-memory JSON-compliant object or list. |
JSON text |
|
JSON, as a string or binary. |
XML text |
|
XML, as a string or binary. Converted to an object, it follows the XML compliant rules. |
Avro, multiple records |
|
Avro binary with one record for each entry of a list. See Avro Conversion. |
Avro, single record |
|
Avro binary with exactly one record, which can be an object or a whole list. |
HDB archive |
|
A copy of Habitat’s database, called HDB. The conversion can read an archive but not write one, so there’s no |
Plain text |
|
A string, with no format of its own. |
Binary |
|
Bytes, with no format of their own. |
A strategy holds one conversion.
Each from… call offers only the to… calls that it can convert to.
Each call’s sub-builder, the block in braces after it, accepts only the options for its own format.
Your IDE suggests only the conversions and options that fit, and the compiler rejects the rest.
The following examples show every conversion.
Each commented example is a separate strategy.
Only Avro conversions require schemaId; all other options are optional.
fromJsonCompliant and toJsonCompliant take no options.
toJsonCompliant passes through a JSON-compliant object or list unchanged.
toPlainText does the same for a string, and toBytes for binary.
Any other payload must be in the from format. Otherwise, the message fails with an error that names both formats and the payload type.
Connect 2.0.0 introduced conversion by format.
Flow-servers on earlier versions ignore the from and to fields and run the processor without a conversion.
|
JSON Conversion
fromJsonText { }.toJsonCompliant() converts JSON text, held as a string or binary, to a JSON-compliant object or list.
The fromJsonText sub-builder accepts:
| Option | Description |
|---|---|
|
The character set the conversion uses to decode JSON text held as bytes. Choose one from Oracle’s list of supported encodings. Defaults to |
// JSON text, as a string or binary, to a JSON-compliant payload
fromJsonText { characterSet = "UTF-16" }.toJsonCompliant()
fromJsonCompliant().toJsonText { } converts a JSON-compliant object or list to JSON text.
The toJsonText sub-builder accepts:
| Option | Description |
|---|---|
|
The character set a processor such as |
|
If |
|
If |
The processor receives JSON text as a string, even when you set characterSet.
A string arriving under toJsonText fails because the conversion reads only a JSON-compliant object or list.
// JSON-compliant payload to JSON text
fromJsonCompliant().toJsonText()
// JSON-compliant payload to indented JSON text
fromJsonCompliant().toJsonText { formattedOutput = true }
// List to newline-delimited JSON (NDJSON) text
fromJsonCompliant().toJsonText { ndJsonOutput = true }
XML Conversion
fromXmlText { }.toJsonCompliant() converts XML text, held as a string or binary, to a JSON-compliant object that follows the XML-compliant formatting rules.
The fromXmlText sub-builder accepts:
| Option | Description |
|---|---|
|
The character set the conversion uses to decode XML text held as bytes. Defaults to |
|
Optional. The IDs of the XSD or WSDL resources that describe the XML, separated by commas, such as |
|
Sets the prefix for a namespace in the object. See Namespace Mapping.
The prefix |
|
If |
// XML text, as a string or binary, to a JSON-compliant object
fromXmlText { characterSet = "ISO-8859-1" }.toJsonCompliant()
// XML with a schema and namespace prefixes to a JSON-compliant object
fromXmlText {
schemaId = "XSD:echo-service:1"
// "_default" names the namespace whose elements get no prefix
namespacePrefix("_default", "http://www.bccs.uib.no/EchoService.wsdl")
namespacePrefix("aa", "http://www.bccs-aa.uib.no/EchoService.wsdl")
}.toJsonCompliant()
// XML with a doctype declaration to a JSON-compliant object
fromXmlText { allowInsecureProcessing = true }.toJsonCompliant()
fromJsonCompliant().toXmlText { } converts an XML-compliant object to XML text.
The toXmlText sub-builder accepts:
| Option | Description |
|---|---|
|
The character set a processor such as |
The processor receives XML text as a string, even when you set characterSet.
A string arriving under toXmlText fails because the conversion reads only an XML-compliant object.
// XML-compliant object to XML text
fromJsonCompliant().toXmlText()
// XML-compliant object to XML text, for a processor that sends it as ISO-8859-1 bytes
fromJsonCompliant().toXmlText { characterSet = "ISO-8859-1" }
Avro Conversion
Avro conversion reads binary into a JSON-compliant payload or writes a JSON-compliant payload as binary.
A string fails under fromAvroRecords and fromAvroSingleRecord.
The two pairs of builders differ in how many records the Avro binary holds.
Multiple Records
toAvroRecords writes one record for each entry of a list, and fails on a payload that is not a list.
fromAvroRecords reads every record into a list, with one entry for each.
Both sub-builders require:
| Option | Description |
|---|---|
|
Required. The ID of one Avro Schema resource, such as |
Use these builders when the binary holds several records, such as several cloud events.
Under toAvroRecords, the processor receives binary.
// Avro binary (multiple records) to a JSON-compliant list
fromAvroRecords {
schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()
// JSON-compliant list to Avro binary (multiple records)
fromJsonCompliant().toAvroRecords {
schemaId = "AvroSchema:mySchema:latest"
}
Single Record
toAvroSingleRecord writes the whole JSON-compliant object or list as one record.
fromAvroSingleRecord reads one record into a JSON-compliant object or list.
Binary with no records or more than one record fails the message.
Both sub-builders require:
| Option | Description |
|---|---|
|
Required. The ID of one Avro Schema resource, such as |
Use these builders when the binary holds one record, such as one cloud event.
Under toAvroSingleRecord, a list payload therefore needs a schema that describes a list.
The processor receives binary.
// Avro binary (single record) to a JSON-compliant object or list
fromAvroSingleRecord {
schemaId = "AvroSchema:mySchema:latest"
}.toJsonCompliant()
// JSON-compliant object or list to Avro binary (single record)
fromJsonCompliant().toAvroSingleRecord {
schemaId = "AvroSchema:mySchema:latest"
}
| Avro conversion does not support the proprietary Confluent wire format. |
HDB Archive Conversion
An HDB archive is a file in which Habitat, a GE Vernova SCADA application, saves a copy of its database (HDB).
fromHdbArchive { }.toJsonCompliant() converts an archive into an object that contains its data.
The fromHdbArchive sub-builder selects the databases, tables, and fields to convert.
It accepts:
| Option | Description |
|---|---|
|
Selects the databases, tables, and fields to convert. If you omit the selection, the conversion includes the whole archive. |
To convert the whole archive, omit the sub-builder, as in fromHdbArchive().toJsonCompliant().
Conversion by format can read an HDB archive but can’t write one.
An archive must arrive as binary, so a string fails under fromHdbArchive.
inboundTransformationStrategy {
fromHdbArchive {
database("BLS") {
table("AREA")
table("LEVEL") { fields("ID_*") }
}
database("BLSLOG") { table("MESS") }
}.toJsonCompliant()
}
In the selection:
-
database(name)converts that database. Without atablesub-builder, it converts all the database’s tables. -
table(name)converts that table. Without afieldscall, it converts all the table’s fields. -
fields(patterns)converts the fields that match the patterns.*matches any characters, soID_*matches every field that starts withID_.
Write database and table names in full.
Only field patterns accept *.
Names are case-sensitive, and Habitat writes them in upper case.
The conversion includes a pseudo-table, such as RL%, only if a table call names it.
The flow spec fails to build, or to read from JSON, if a database or table name contains * or if a name appears twice in one sub-builder.
The selection filters columns, not rows. Each table includes all its rows. Each row also keeps the columns that place it in the table tree, described in the list below. To keep only some rows, add a processor after the conversion to filter them.
A name or pattern that matches nothing in the archive fails the message.
The error lists what the archive holds.
For example, table("ITEM") in an archive whose database ALARMDEF holds the table ITEMS fails with:
the payload holds no table matching ITEM in the databases matching ALARMDEF; those hold [ITEMS, RL%, RH%, RP%]
The resulting object contains databases, with one entry for each converted database:
{
"databases": [
{
"name": "LOGMAN",
"tables": [
{
"name": "QUEUE",
"fields": ["subscript", "parentSubscript", "MAX_QUEUE", "NAME_QUEUE"],
"data": [[1, 1, 3000, "EVENTQ"], [2, 1, 20, "FILEQ"]],
"fieldsWithoutData": []
}
],
"tableTree": [
{ "table": "DECK", "parentTable": null, "level": 2 },
{ "table": "QUEUE", "parentTable": "DECK", "level": 3 }
]
}
]
}
-
tablescontains the converted tables. Without a selection, that is every table with data, except the pseudo-tables that Habitat keeps for itself.fieldsnames a table’s columns once.datacontains one list of values for each row, in the order thatfieldslists them. -
Habitat arranges its tables in a tree, and a row of one table can belong to a row of the table above it. The first columns of each row give its place in that tree.
subscriptis the row’s number in its table, counting from one.parentSubscriptis the number of its parent row, where the table hangs below another. -
tableTreelists every table in the database, pseudo-tables included, with the table each hangs below. It always lists them all, whatever the selection names. -
A null in
datais either a value the database does not hold, or a float that is NaN or infinite. The two look the same.fieldsWithoutDatanames the fields the archive holds no data for.
Two things differ from what Habitat’s own tools print:
-
An array field comes back as a list indexed from zero. Habitat’s tools number the same elements from one.
-
A timestamp comes back in UTC, as an ISO-8601 instant such as
2002-06-26T19:25:14Z. Habitat’s tools print it in the time zone they are set to.
An inboundTransformationStrategy does not support streamed payloads, so keep streamingEnabled off in a flow that reads archives with readFiles.
A payload that is not an archive, or is damaged, fails the message.
|
An archive usually contains compressed data, which can grow to many times the archive’s size during decompression. The conversion streams the decompressed archive into a temporary file, then loads the selected data into memory. A large selection can take more memory than the flow-server has. Select only the data the flow needs.
Convert a whole archive only when the flow uses all of its data.
Throttle the flow with |
The conversion writes the temporary file to the Java temporary directory (java.io.tmpdir) unless an operator sets a different directory with mc.flow-server.hdb-archive.temp-directory in the flow-server configuration.
Plain Text Conversion
fromBytes { }.toPlainText() decodes binary to a plain-text string.
toPlainText takes no options.
The fromBytes sub-builder accepts:
| Option | Description |
|---|---|
|
The character set the conversion uses to decode the bytes. Defaults to |
// Binary to plain text
fromBytes().toPlainText()
Binary Conversion
fromPlainText().toBytes { } encodes a plain-text string as binary.
fromPlainText takes no options.
The toBytes sub-builder accepts:
| Option | Description |
|---|---|
|
The character set the conversion uses to encode the text as bytes. Defaults to |
// Plain text to binary
fromPlainText().toBytes { characterSet = "ISO-8859-1" }