Contact Us 1-800-596-4880
  1. Homepage
  2. Design Center
  3. Flow Designer
  4. Add Logic, Data Transformations, Error Catching, and More to Flows

  5. Process Elements in a Collection

Process Elements in a Collection

You can process elements in a collection that is passed in a payload to an instance of the For-Each component or of the Parallel For-Each component. Each component sets a scope within which you can add other types of component, as well as connectors and modules, for processing each element.

Two simple examples: one of the use of For-Each
Figure 1. The For-Each component and the Parallel For-Each component both set scopes for processing the elements within a collection.

  1. The For-Each component in this example inserts each element, one at a time, into a database. Each element is presumably a string that can be used in a SQL INSERT statement.

  2. The Parallel For-Each component in this example sends each element to a Set Payload component that modifies the element with an expression and adds it to the flow’s payload.

Both types of component can process any collection.

How the For-Each Component Processes Data and Continues a Flow

The For-Each component is for processing each element in a collection serially and then proceeding in the flow with the original payload that was passed to the For-Each component:

Diagram depicting the actions taken by the For-Each component
Figure 2. This diagram represents a flow in which a For-Each component is used. The rounded rectangles are cards.

  1. A card passes the payload of a flow to the For-Each component. The payload consists of a collection of elements.

  2. The For-Each component divides the collection into separate elements. It then sends each element serially to the first component, connector, or module that is in its scope. Each element is treated as a payload within the scope. So, if a connector within the scope is configured to act on payload or uses a script to process payload, the payload is not the payload that the For-Each component received but the element that the component is processing.

  3. After the final element is processed, the For-Each component passes the payload that it received from the previous card to the next card in the flow.

  4. The next card in the flow receives the same payload that the For-Each component received, and the flow continues.

How the Parallel For-Each Component Processes Data and Continues a Flow

The Parallel For-Each component is for processing each element in a collection and then passing the results to the next card in the flow:

Diagram depicting the actions taken by the Parallel For-Each component
Figure 3. This diagram represents a flow in which a Parallel For-Each component is used. The rounded rectangles are cards.

  1. A card passes the payload of a flow to the Parallel For-Each component. The payload consists of a collection of elements.

  2. The Parallel For-Each component divides the collection into separate elements. It then sends elements simultaneously to the first component, connector, or module that is in its scope. You can specify the maximum number of elements to process at a time. Each element is treated as a payload within the scope. So, if a connector within the scope is configured to act on payload or uses a script to process payload, the payload is not the payload that the Parallel For-Each component received but the element that the component is processing.

  3. After all of the elements are processed, the Parallel For-Each component aggregates and passes the results of the processing that was done within its scope. The results form the payload that is passed to the next card in the flow.

  4. The next card in the flow receives the results from the Parallel For-Each component, and the flow continues.

The order of the results correspond to the order of the elements in the original payload. For example, suppose that the payload that is passed to a Parallel For-Each component is ["Apple","Banana","Cherry"]. Within the component’s scope is a Set Payload component that processes each element as a separate payload. The Set Payload component processes each payload with this DataWeave expression: payload ++ '-result'. The Parallel For-Each component passes these results to the next card in the flow: ["Apple-result","Banana-result","Cherry-result"].

Within the scope of the Parallel For-Each component, the order in which each element is processed is not guaranteed to be the order in which each element appears in a collection. Therefore, if any of the connectors require elements in a particular order (for example, if you want to update database tables in a particular order with the Database connector), you should use the For-Each component instead.

Moreover, you should be aware whether any APIs or services that you want to connect to within the scope of a Parallel For-Each component have any limits in the number of concurrent connections that they accept. If they do, you must specify a maximum number of concurrent processes for the Parallel For-Each component.

Batching

With the For-Each component, you can split a collection into batches to enable quicker processing. Each batch is treated as a separate Mule message. For example, if a collection has 200 elements and you set Batch Size to 50, the For-Each component iteratively processes 4 batches of 50 elements, each as a separate Mule message.

Batching is not supported with the Parallel For-Each component.

Variables

Every iteration within the For-Each component can use the variables that were used in the previous iteration. New variables or modifications to existing variables that take place during the processing of one element are visible during the processing of another element. These changes to variables continue to be available outside the For-Each scope.

For example, suppose that you use a Set Variable component just before the scope of a For-Each component. Within the Set Variable component, you set the variable NumOfElements to 0. You then use a Set Variable component in the For-Each scope, using the same variable, that specifies a DataWeave script that increments the value by 1 every time it is called. Every time that the For-Each component sends another element of a collection to be processed, the value is incremented and saved in NumOfElements. When all of the elements have been processed, the value of NumOfElements is the number of elements that were in the collection. The value can be used outside of the For-Each scope by subsequent cards in the flow.

For the Parallel For-Each component, the value of a variable that enters the scope is the same at the start of the processing of each variable. New variables or modifications of already existing variables during the processing of one element are not visible during the processing of another element. Moreover, no changes to variables with the scope of a Parallel For-Each component are available to the subsequent cards in the flow.

Suppose that, as in the previous example, you use a Set Variable component just before the scope of a Parallel For-Each component. Within the Set Variable component, you set the variable NumOfElements to 0. You then use a Set Variable component in the Parallel For-Each scope, using the same variable, that specifies a DataWeave script that increments the value by 1 every time it is called. For each parallel process that the component runs, the highest value reached by NumOfElements is 1. And if subsequent cards in the flow use NumOfElements, its initial value after the Parallel For-Each component finishes processing the collection is 0, which is the same value it had before the Parallel For-Each component started.

Error Handling

Both the For-Each component and Parallel For-Each component stop processing a collection if an element causes any component, connector, or module in a scope to throw an exception. In addition, the Parallel For-Each component stops processing a collection if one of the parallel processes times out while waiting for an open connection to a system. You can set the maximum length of time that a parallel process can wait.

When the processing of a collection stops because of an exception, Flow Designer displays a red circle on the card that threw the exception. When the processing of a collection stops because of a timeout, Flow Designer displays a red circle on the Parallel For-Each component’s card. You can open the Logs pane to read the error message.

Configuring a For-Each Component

When you configure this component, you must specify values in these fields:

Collection

An expression that returns a Java collection, object array, map, or DOM nodes. The default is payload.

Batch Size

Partitions the collection into subcollections of the specified number of elements. For example, if a collection has 200 elements and you set the batch size to 50 elements, the component processes 4 batches of 50 elements each.

Root Message Variable Name

Name of the variable that stores the entire message (payload, variables, and attributes) that was passed to a For-Each component from the previous card in a flow.

Counter Variable Name

Name of the variable that stores the index of the element that is being processed. For example, if a collection is ['a','b','c'], the index of a is 0, of b is 1, and of c is 2. When the For-Each component is processing the second element, the value of the variable is 1.

Configuring a Parallel For-Each Component

When you configure this component, you must specify values in these fields:

Collection

Specifies the expression that defines the collection of parts to be processed in parallel. The default is payload.

Timeout

Specifies the maximum time in milliseconds that the entire scope of a Parallel For-Each component can take to complete processing an element in a collection.

For example, within a scope you might place an HTTP Request component that sends a GET request to a service and an instance of Database Connector for inserting values into a database. Each element in your collection specifies what to request in the GET request. The value of Timeout specifies how long to wait in milliseconds for both the service to return the information requested and for the instance of Database Connector to insert the information into the database.

Max Concurrency

Specifies the maximum number of parallel processes that can be run. How many parallel processes are in fact started depends on how Mule Runtime allocates resources where it is running:

  • If you set the value to 40, but Mule Runtime has allocated only enough resources to run up to 4 processes in parallel, only up to 4 processes can run in parallel.

  • If you set the value to 2, but Mule Runtime has allocated enough resources to run up to 4 processes in parallel, only up to 2 processes can run in parallel

When you configure this component, you can, but are not required to, specify values in these fields:

Target

Specifies a variable to use for storing the results of processing all of the elements in a collection. By default, the output is saved in the flow’s payload.

Target Value

Specifies an expression to evaluate against the operation’s output value. The outcome of this expression is stored in the target variable. By default, this is the same as the operation’s output value.