In this article
Use this Snap to run multiple Pipelines through one Pipeline.
This Snaps supports the following modes of data processing:
This Snap replaces the ForEach and Task Execute Snaps, as well as the Nested Pipeline mechanism. |
Works in Ultra Task Pipelines with the following exception: Reusing a runtime on another Snaplex is not supported. See Snap Support for Ultra Pipelines.
Supports child Pipelines in eXtreme mode. This Snap essentially functions as a bridge between Standard and eXtreme modes. See the example for more information.
None
If there are not enough Snaplex nodes to execute the Pipeline on the Snaplex, then the Snap waits until Snaplex resources are available. When this situation occurs, the following message appears in the execution statistics dialog:
Because a large number of Pipeline runtimes can be generated by this Snap, only the last 100 completed child Pipeline runs are saved for inspection in the Dashboard.
The Input/Output views on the Views tab are configurable.
Input/Output | Type of View | Number of Views | Examples of Upstream and Downstream Snaps | Description | |
---|---|---|---|---|---|
Input | Binary or Document |
|
| The document or binary data to send to the child Pipeline. | |
Output | Binary or Document |
|
|
| |
Error | N/A | N/A | N/A | This Snap has at most one document error view and produces zero or more documents in the view. If the Pipeline execution does not complete successfully, a document is written to this error view. |
Parameter Name | Data Type | Description | Default Value | Example | |
---|---|---|---|---|---|
Label | String | N/A | ExecuteCustomerUpdatePipeline | ||
Pipeline | String | Required. Enter the absolute or relative (expression-based) path to the child Pipeline to run. If you only enter the Pipeline name, then the Snap searches for the Pipeline in the following folders in this order:
You can specify the absolute path to the target project using the Org/project_space/project notation. You can also dynamically choose the Pipeline to run by entering an expression in this field when Expression enabled. For example, to run all of the Pipelines in a project, you can connect the SnapLogic List Snap to this Snap to retrieve the list of Pipelines in the project and run each one. | N/A | NetSuite-Create-Credit-Memo | |
Execute On | String | Select the one of the following Snaplex options to specify the target Snaplex for the child Pipeline:
Best PracticesWhen choosing which Snaplex to run, always consider first the LOCAL_NODE option. With this selection, the child Pipeline runtime occurs on the same Snaplex node as that of the parent Pipeline. Due to absence of network traffic required to start the child Pipeline on another node, having the runtime on the same node makes the execution faster and more reliable. In both of these cases, you do not need to set an Org-specific property. This scenario facilitates usage for the user importing the Pipeline. However, if you do need to run the child Pipeline on a different Snaplex than the parent Pipeline, then select SNAPLEX_WITH_PATH and enter the target Snaplex name. In this case, you can take advantage of the following strategies:
| SNAPLEX_WITH_PATH | groundplex4-West | |
Snaplex Path | String | Required when you select the SNAPLEX_WITH_PATH option in the Execute On field. Enter the name of the Snaplex on which you want the child Pipeline to run. Click to select from the list of Snaplex instances available in your Org. | N/A | DevPlex-1 | |
Execution label | String | The label to display in the Pipeline view of the Dashboard. You can use this field to differentiate one Pipeline execution from another. | The label for the child Pipeline. | NetSuite-Create-Credit-Memo | |
Pipeline Parameters | Specify the Pipeline Parameters for the Pipeline selected in the Pipeline field. Click to add a row and enter key and value pairs for the following fields:
The Parameter Value can be an expression that you configure based on incoming documents, or the value can be a constant. When you select Reuse executions to process documents, you cannot change parameter values from one Pipeline invocation to the next. | ||||
Parameter name | String | Enter the name of the parameter. You can select the Pipeline Parameters defined for the Pipeline selected in the Pipeline field. | N/A | Postal_Code | |
Parameter value | String | Enter the value for the Pipeline Parameter, which can be an expression based on incoming documents or a constant. If you configure the value as an expression based on the input, then each incoming document or binary data is evaluated against that expression when invoking the Pipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next. | N/A | 94402 | |
Reuse executions to process documents | Checkbox | Select this option to start a child Pipeline and pass multiple inputs to the Pipeline. Reusable executions continue to live until all of the input documents to this Snap have been fully processed. When you enable this option and the Pipeline parameters use expressions, the expressions are evaluated using the first document. The parameter value in the child Pipeline does not change across documents. If you do not select this option, then a new Pipeline execution is created for each input document. | Not selected | Selected | |
Number of Retries | String | Enter the maximum number of retry attempts that the Snap must make in case of a network failure. If the child Pipeline does not execute successfully, an error document is written to the error view.
| 0 | 3 | |
Retry Interval | String | Enter the minimum number of seconds for which the Snap must wait between two successive retry requests. A retry happens only when the previous attempt resulted in an error. | 1 | 10 | |
Pool size | String | Enter multiple input documents or binary data to be processed concurrently by specifying an execution pool size. When the pool size is greater than one, then the Snap starts Pipeline executions as needed for up to the specified pool size. When you select Reuse executions to process documents, the Snap starts a new execution only if all executions are busy working on documents or binary data and the total number of executions is below the pool size. | 1 | 4 | |
Timeout | String | Enter the maximum number of seconds for which the Snap must wait for the child Pipeline to complete the runtime. If the child Pipeline does not complete the runtime before the timeout, the execution process stops and is marked as failed. | No timeout is the default | 10 | |
Execute and Validate | Disabled |
No unlinked binary input or output views. When you enter an expression in the Pipeline field, the Snap needs to contact the SnapLogic cloud servers to load the Pipeline runtime information. Also, if Reuse executions to process documents is enabled, the result of the expression cannot change between documents.
The Pipeline Execute Snap can replace some uses of the Nested Pipeline mechanism as well as the ForEach and Task Execute Snaps. For now, this Snap only supports child Pipelines with unlinked document views (binary views are not supported). If these limitations are not a problem for your use case, read on to find out how you can transition to this Snap and the advantages of doing so.
Converting a Nested Pipeline will require the child Pipeline to be adapted to have no more than a single unlinked document input view and no more than a single unlinked document output view. If the child Pipeline can be made compatible, then you can use this Snap by dropping it on the canvas and selecting the child Pipeline for the Pipeline property. You will also want to enable the Reuse property to preserve the existing execution semantics of Nested Pipelines. The advantages of using this Snap over Nested Pipelines are:
Converting a ForEach Snap to the Pipeline Execute Snap is pretty straightforward since they have a similar set of properties. You should be able to select the Pipeline you would like to run and populate the parameter listing. The advantages of using this Snap over the ForEach Snap are:
Converting a Task Execute Snap to the Pipeline Execute Snap is also straightforward since the properties are similar. To start, you only need to select the Pipeline you would like to use, you no longer have to create a Triggered Task. If you set the Batch Size property in the Task Execute to one, then you will not want to enable the Reuse property. If the Batch Size was greater than one, then you should enable Reuse. The Pipeline parameters should be the same between the Snaps. The advantages of using this Snap over the Task Execute Snap are:
Converting a Pipeline that uses the Router Snap in "auto" mode can be done by moving the duplicated portions of the Pipeline into a new Pipeline and then calling that Pipeline using a Pipeline Execute. After refactoring the Pipeline, you can adjust the "Pool Size" of the Pipeline Execute Snap to control how many operations are done in parallel. The advantages of using this Snap over an "Auto" Router are:
You can now browse the Pipeline Catalog for the target child Pipeline, and then select, drag and drop it in the Canvas. The SnapLogic Designer automatically adds the child Pipeline using a Pipeline Execute Snap.
Likewise, you can preview a child Pipeline by hovering over a Pipeline Execute Snap while the parent Pipeline is open on the Designer Canvas.
A common use case for the Pipeline Execute Snap is to run a child Pipeline whose output is immediately returned to the parent Pipeline for further processing. You can achieve this return with the following Pipeline design for the child Pipeline.
In this example, the document in this child Pipeline is sent to the parent Pipeline through output1 of the Copy Snap. Any unconnected output view is returned to the parent Pipeline. You can use any Snap that completes execution in this way.
This example Pipeline uses a Pipeline Execute Snap in standard-mode to call and execute an eXtreme-mode Pipeline as a child Pipeline.
This eXtreme-mode Pipeline (azd - numbers) reads a CSV file from Azure Storage blob (WASB), parses the file in CSV format, transforms the data using certain calculations in Aggregate - SparkSQL 2.x Snap, formats the CSV output file, and finally writes the file to the specified path.
In this Pipeline, we configure the Pipeline Execute Snap to call the eXtreme-mode Pipeline. Hence, we specify the child Pipeline name azd - numbers and the Snaplex Path azd_DS3_v2_1w on which to execute the eXtreme-mode Pipeline.
Upon running the Pipeline Execute Snap (standard-mode), it locates and successfully executes the selected eXtreme-mode Pipeline. We can view the statistics of the Pipeline execution as shown below:
Download this Pipeline.
The PE_Multiple_Executions project demonstrates how you can configure the Pipeline Execute Snap to execute a child Pipeline multiple times. The project contains the following Pipelines:
The project, PE_Backward_Schema_Propagation_Contacts, demonstrates the schema suggest feature of the Pipeline Execute Snap. It contains the following files:
The parent Pipeline is shown below:
The child Pipeline is as shown below:
The Pipeline Execute Snap is configured as:
The following schema is provided in the JSON Formatter Snap. It has three properties - $firsName, $lastName, and $age. This schema is back propagated to the parent Pipeline.
The parent Pipeline must be validated in order for the child Pipeline's schema to be back-propagated to the parent Pipeline. Below is the Mapper Snap in the parent Pipeline:
Notice that the Target Schema section shows the three properties of the schema in the child Pipeline:
Upon execution the data passed in the Mapper Snap will be written into the test.json file in the child Pipeline. The exported project is available in the Downloads section below.
The project, PE_Backward_Forward_Schema_Propagation, demonstrates the Pipeline Execute Snap's capability of propagating schema in both directions – upstream as well as downstream. It contains the following Pipelines:
|