Resource Registry

The Utilihive Resource Registry is a collection of schemas, documents, credentials, etc. that certain flow processors depend on. For example, a restApi processor would need to reference a separate specifications file to establish its REST endpoints. That file would reside in the Resource Registry.

Items in the Resource Registry are classified as either a "resource" (e.g., a schema), a "secret" (e.g., credentials), or a "table". Resources and secrets can be managed through the Customers Management API, the SDK Deployer, or Utilihive Heartbeat. Tables are currently only accessible through the Customers Management API and processor scripts.

Resources

Processors that depend on a resource/schema simply contain a reference ID, and the flow-server will look up the matching resource in the Resource Registry. The following example establishes a restApi processor that points to an OpenAPI specification resource:

restApi {
    id = "echo-api"
    apiSpecId = "OpenAPIv3:simple-rest-api:latest"
}

Resources are referenced on the format <type>:<resourceId>:<revision>, where resourceId is a custom-defined name. revision is either a revision number (e.g., 1) or the string latest.

Types

The following resource types are available:

Type Description

ArrowSchema

Apache Arrow Schema specification written in JSON.

AvroSchema

Apache Avro Schema specification written in JSON.

JsonSchema2019-09

Draft 2019-09 compliant JSON validation schema that doesn’t refer to external validation schemas.

OpenAPIv3

OpenAPI REST interface specification written in JSON or YAML.

WSDLv1

WSDL document with web service specifications.

XSD

XML schema document that doesn’t import or include other XSDs.

Arrow Schema Requirements

Processors that read or write Arrow data, will require an Arrow Schema that defines the structure of the Arrow data. As Arrow is a columnar format, the schema will specify the data type of each column. Each column specification is called a field in Arrow Schema.

The specification of supported fields and how each field is specified in JSON is described in full in the Arrow Schema section.

Avro Schema Requirements

Processors that read or write Avro data, will require an Avro Schema that defines the structure of the Avro data. The Avro Schema is used for the serialisation into binary Avro format and subsequent de-serialisation of the binary data.

A guideline for writing Avro schemas is found here.

WSDLv1 Requirements

WSDL documents must satisfy the following criteria:

  • Version 1.1 (version 2.0 is not supported)

  • Uses SOAP as web service protocol

  • Uses only a one-way or request-response operation type

  • The only supported message style is Document/literal. RPC/encoded, RPC/literal and Document/encoded are not supported

  • Each <message> element has exactly one <part> element

  • Has unique operation names and operation input types (inside each <portType> element)

  • No imports of other documents (like external XSDs)

    Use the SDK’s merge tool to consolidate imported documents.

The processor will throw a ValidationException if one of the following occurs:

  • Parsing of WSDL document fails (malformed document, wrong version, contains imports, etc.)

  • Document doesn’t contain required parts (<message>, <portType>, <binding>, etc.)

  • WSDL doesn’t specify SOAP-related data

Secrets

Processors that require credentials to integrate with a protected system only need to reference an auth configuration key. The flow-server then uses this key to look up the actual secret/credentials in the Resource Registry. For example, the following restRequest processor references a secret with the key backendAuth:

restRequest {
    id = "rest-request"
    address = URL("https://www.test.com/api")
    authenticationConfigKey = "backendAuth"
}

The credentials themselves are key/value pairs that might look like the following:

userName=orderAdmin
password=myPassword

Note that userName and password are a recognized format in Utilihive’s authentication system. See the Secret Types documentation for other supported formats.

Dynamic Tables

Dynamic tables are a data structure that can be accessed by all flows under the same organization. A dynamic table consists of a list of rows containing a list of cells, where each cell is associated with a column name. The structure of the table (i.e., the columns) is entirely up to the user and can be dynamically altered during lookup operations.

Scripting processors like map can use built-in functions to retrieve and/or create table rows. For example, the following mapping uses the incoming payload to look up and return a postal code from a postCodes table:

let #row := tableRowLookup("postCodes", "city", #input.payload)
return #row.code

See the Dynamic Tables documentation for more details on how to use and test dynamic tables.