DataWeave Output Formats and Writer Properties

In some cases, it is necessary to modify or specify aspects of the format through format-specific properties. For example, you can specify CSV input and output properties, such as the separator (or delimiter) to use in the CSV file. For Cobol copybook, you need to specify the path to a schema file using the schemaPath property.

You can append reader properties to the MIME type (outputMimeType) attribute for certain components in your Mule app. Listeners and Read operations accept these settings. For example, this On New File listener example identifies the , separator for a CSV input file:

Note that the outputMimeType setting above helps the CSV reader interpret the format and delimiter of the input comma_separated.csv file, not the writer.

To specify the output format, you can provide the MIME type and any writer properties for the writer, such as the CSV or JSON writer used by a File Write operation. For example, you might need to write a pipe (|) delimiter in your CSV output payload, instead of some other delimiter used in the input. To do this, you append the property and its value to the output directive of a DataWeave expression. For example, this Write operation specifies the pipe as a separator:

The sections below list the format-specific reader and writer properties available for each supported format.

When specifying application/avro as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

When defining application/avro input for the DataWeave reader, you can set the following property.

When you import a Copybook definition, the Transform Message component converts the definition to a flat file schema that you can reference with schemaPath property.

To import a copybook definition:

For example, assume that you have a copybook definition file (mailing-record.cpy) that looks like this:

When you import the schema, the Transform component converts the copybook file to a flat file schema that it stores in the src/main/resources/schema folder of your Mule project. In flat file format, the copybook definition above looks like this:

After importing the copybook, you can use the schemaPath property to reference the associated flat file through the output directive. For example: output application/flatfile schemaPath="src/main/resources/schemas/mailing-record.ffd"

Not all copybook features are supported by the Cobol Copybook format in DataWeave. In general, the format supports most common usages and simple patterns, including:

Unsupported features include:

REDEFINES facilitates dynamic interpretation of data in a record. When you import a copybook with REDEFINES present, the generated schema uses a special grouping with the name '*' (or '*1', '*2', and so on, if multiple REDEFINES groupings are present at the same level) to combine all the different interpretations. You use this special grouping name in your DataWeave expressions just as you use any other grouping name.

Use of REDEFINES groupings has higher overhead than normal copybook groupings, so MuleSoft recommends that you remove REDEFINES from your copybooks where possible before you import them into Studio.

BINARY (COMP), COMP-5, or PACKED-DECIMAL (COMP-3) usages are only supported with single-byte character encodings, which use the entire range of 256 potential character codes. UTF-8 and other variable-length encodings are not supported for these usages (because they’re not single-byte), and ASCII is also not supported (because it doesn’t use the entire range). Supported character encodings include ISO-8859-1 (an extension of ASCII to full 8 bits) and other 8859 variations and EBCDIC (IBM037).

REDEFINES requires you to use a single-byte-per-character character encoding for the data, but any single-byte-per-character encoding can be used unless BINARY (COMP), COMP-5, or PACKED-DECIMAL (COMP-3) usages are included in the data.

The most common issue with copybook imports is a failure to follow the Cobol standard for input line regions. The copybook import parsing ignores the contents of columns 1-6 of each line, and ignores all lines with an '*' (asterisk) in column 7. It also ignores everything beyond column 72 in each line. This means that all your actual data definitions need to be within columns 8 through 72 of input lines.

Tabs in the input are not expanded because there is no defined standard for tab positions. Each tab character is treated as a single space character when counting copybook input columns.

Indentation is ignored when processing the copybook, with only level-numbers treated as significant. This is not normally a problem, but it means that copybooks might be accepted for import even though they are not accepted by Cobol compilers.

Both warnings and errors might be reported as a result of a copybook import. Warnings generally tell of unsupported or unrecognized features, which might or might not be significant. Errors are notifications of a problem that means the generated schema (if any) will not be a completely accurate representation of the copybook. You should review any warnings or errors reported and decide on the appropriate handling, which might be simply accepting the schema as generated, modifying the input copybook, or modifying the generated schema.

When defining application/flatfile input for the DataWeave reader, you can set the properties described in Reader Properties (for Flat File).

Note that schemas with type Binary or Packed don’t allow for the detection of line breaks, so setting recordParsing to lenient only allows for long records to be handled, not short ones. These schemas only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

When specifying application/flatfile as the output format in a DataWeave script, you can add the properties described in Writer Properties (for Flat File) to change the way the DataWeave parser processes the data.

In CSV, you can assign any special character as the indicator for separating fields, toggling quotes, or escaping quotes. Make sure you know what special characters are in your input so that DataWeave can interpret them correctly.

When defining application/csv input for the DataWeave reader, you can set the following property.

By default, the CSV reader stores input data from an entire file in-memory if the file is 1.5MB or less. If the file is larger than 1.5 MB, the process writes the data to disk. For very large files, you can improve the performance of the reader by setting a streaming property to true. To demonstrate the use of this property, the next example streams a CSV file and transforms it to JSON.

Note that the example indicates that the input file is located in Studio project directory src/main/resources, which is the location of ${app.home}.

The structure of the CSV input looks something like the following.

Note that a streamed file is typically much longer.

When specifying application/csv as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

All of these parameters are optional.

A CSV output directive example might look like this:

In the Transform component, you can define a CSV type through the following methods:

When specifying application/dw as the output format in a DataWeave script, you can add the following properties to change the way the parser processes data.

When defining application/xlsx input for the DataWeave reader, you can set the following property.

By default, the Excel reader stores input data from an entire file in-memory if the file is 1.5MB or less. If the file is larger than 1.5 MB, the process writes the data to disk. For very large files, you can improve the performance of the reader by setting a streaming property to true. To demonstrate the use of this property, the next example streams a XLSX file and transforms it to JSON.

The following example streams an Excel file and transforms it to JSON.

The example:

When specifying application/xlsx as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

All of these parameters are optional. A DataWeave output directive for Excel might look like this:

In the Transform component, you can define a Excel type through the following method:

When defining application/flatfile input for the DataWeave reader, you can set the properties described in Reader Properties (for Flat File).

Note that schemas with type Binary or Packed don’t allow for the detection of line breaks, so setting recordParsing to lenient only allows for long records to be handled, not short ones. These schemas only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

When specifying application/flatfile as the output format in a DataWeave script, you can add the properties described in Writer Properties (for Flat File) to change the way the DataWeave parser processes the data.

All of the properties are optional.

A DataWeave output directive might look like this:

In the Transform component, you can define a Fixed Width type through the following methods:

When defining application/flatfile input for the DataWeave reader, you can set the following property.

Note that schemas with type Binary or Packed don’t allow for line break detection, so setting recordParsing to lenient only allows long records to be handled, not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

When specifying application/flatfile as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

In the Transform component, you can define a Flat File type by pointing to a schema file.

When defining multipart/form-data input for the DataWeave reader, you can set the following property.

You can set the boundary for the reader to use when it analyzes the data.

Note that in the DataWeave read function, you can also pass the property as an optional parameter. The scope of the property is limited to the DataWeave script where you call the function.

When specifying multipart/form-data as the output format in a DataWeave script, you can add the following property to change the way the DataWeave parser processes data.

In the output directive, you can also set a property for the writer to use when it outputs the data in the specified format.

For example, if a boundary is 34b21, then you can pass this:

Note that in the DataWeave write function, you can also pass the property as an optional parameter. The scope of the property is limited to the DataWeave script where you call the function.

When specifying application/java as the `output format in a DataWeave script, you can add the following property to change the way the DataWeave parser processes data.

There are a couple of custom Java types:

In the Transform component, you can define a Java type through the following method:

When specifying application/json as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

When defining application/json input for the DataWeave reader, you can set the following property.

To demonstrate streaming, the following example streams a JSON file by reading each element in an array one at a time.

The JSON input payload looks like the following.

In the Transform component, you can define a JSON type through the following methods:

When specifying application/x-ndjson as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

When defining application/x-ndjson input for the DataWeave reader, you can set the following property.

When specifying application/octet-stream as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

When specifying text/plain as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

When defining text/x-java-properties output in the DataWeave output directive, you can change the way the parser behaves by adding optional properties.

When defining application/xml input for the DataWeave reader, you can set the following properties.

When specifying application/xml as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

The inlineCloseOn parameter defines whether the output is structured like this (the default):

It can also be structured like this (set with a value of empty):

See also, Example: Outputting Self-closing XML Tags.

In the Transform component, you can define a XML type through the following methods:

MIME Type: application/xml

CData is a custom data type for XML that is used to identify a CDATA XML block. It can tell the writer to wrap the content inside CDATA or to check if the input string arrives inside a CDATA block. CData inherits from the type String.

When specifying application/x-www-form-urlencoded as the `output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.

Note that in the DataWeave write function, you can also pass the property as an optional parameter. The scope of the property is limited to the DataWeave script where you call the function.

When specifying application/yaml as the output format in a DataWeave script, you can add the following properties to change the way the DataWeave parser processes data.