In this article

Overview

Use this Snap to run multiple Pipelines through one Pipeline. 

Supported Data Processing Modes

This Snaps supports the following modes of data processing:

• Run a Pipeline for every input document. Give the target Pipeline an unlinked input view, so that the input document to this Snap is passed into the unlinked input of the child Pipeline. Similarly, if you give the target Pipeline an unlinked output view, then the output document from the child Pipeline is used as the output document from the Pipeline Execute Snap.
• Run one or more Pipelines and reuse them to process multiple input documents to this Snap. Give the child Pipeline one unlinked input view and one unlinked output view. As documents are received by this Snap, they are passed to the child Pipeline executions for processing. The output documents from a child Pipeline are used as the output of this Snap.
• Run Pipelines on the same Snaplex node that the parent Pipeline is running without contacting the SnapLogic cloud. In this mode, the Snap does not need to connect with the SnapLogic cloud servers, allowing you to use this Snap in an Ultra Pipeline. For example, you can use Snaps that do not support Ultra mode with this design.

Supported Pipeline Modes 

• Works in Ultra Pipelines with the following exception: Reusing a runtime on another Snaplex is not supported. See Snap Support for Ultra Pipelines.

• Supports child Pipelines in eXtreme mode. This Snap essentially functions as a bridge between Standard and eXtreme modes.

Prerequisites

None

Limitations and Known Issues

• If there are not enough Snaplex nodes to execute the Pipeline on the Snaplex, then the Snap waits until Snaplex resources are available. When this situation occurs, the following message appears in the execution statistics dialog:
  

• Because a large number of Pipeline runtimes can be generated by this Snap, only the last 100 completed child Pipeline runs are saved for inspection in the Dashboard.

Snap Input and Output

The Input/Output views on the Views tab are configurable.

Snap Settings

Guidelines for Child Pipelines

• No unlinked binary input or output views. When you enter an expression in the Pipeline field, the Snap needs to contact the SnapLogic cloud servers to load the Pipeline runtime information. Also, if Reuse executions to process documents is enabled, the result of the expression cannot change between documents.

• When reusing Pipeline executions, there must be one unlinked input view and zero or one unlinked output view.
• If you do not enable Reuse executions to process documents, then use at most one unlinked input view and one unlinked output view.
• If the child Pipeline has an unlinked input view, make sure the Pipeline Execute Snap has an input view because the input document or binary data is fed into the child Pipeline.
• If you rename the child Pipeline, then you must manually update the reference to it in the Pipeline field; otherwise, the connection between child and parent Pipeline breaks. 
• The child Pipeline is executed in preview mode when the Pipeline containing the Pipeline Execute Snap is saved. Consequently, any Snaps marked not to execute in preview mode are not executed and the child Pipeline only processes 50 documents. 
• You cannot have a Pipeline call itself: recursion is not supported.

Schema Propagation Guidelines

• If the child Pipeline has a Snap that supports schema suggest, such as the JSON Formatter Snap or MySQL Insert Snap, then the schema is back propagated to the parent Pipeline. This configuration is useful in mapping input values in the parent Pipeline to the corresponding fields in the child Pipeline using a Mapper Snap in the parent Pipeline. 
• For the schema suggest to work, the parent Pipeline must be validated first; only then is the schema from the child Pipeline visible in the parent Pipeline.
• The Pipeline Execute Snap is capable of propagating schema in both directions: upstream as well as downstream. See the example Schema Propagation in Parent Pipeline – 3.

Ultra Mode Compatibility

• If you selected Reuse executions to process documents for this Snap in an Ultra Pipeline, then the Snaps in the child Pipeline must also be Ultra-compatible.
• If you need to use Snaps that are not Ultra-compatible in an Ultra Pipeline, you can create a child Pipeline with those Snaps and use a Pipeline Execute Snap with Reuse executions to process documents disabled to invoke the Pipeline. Since the child Pipeline is executed for every input document, the Ultra Pipeline restrictions do not apply.
  For example, if you want to run an SQL Select operation on a table that would return more than one document, you can put a Select Snap followed by a Group By N Snap with the group size set to zero in a child Pipeline. In this configuration, the child Pipeline performs the select operation during execution, and then the Group By Snap gathers all of the outputs into a single document or as binary data. That single output document or binary data can then be used as the output of the Ultra Pipeline.

Migrating from Legacy Nested Pipelines

The Pipeline Execute Snap can replace some uses of the Nested Pipeline mechanism as well as the ForEach and Task Execute Snaps. For now, this Snap only supports  child Pipelines with unlinked document views (binary views are not supported). If these limitations are not a problem for your use case, read on to find out how you can transition to this Snap and the advantages of doing so.

Nested Pipeline

Converting a Nested Pipeline will require the child Pipeline to be adapted to have no more than a single unlinked document input view and no more than a single unlinked document output view. If the child Pipeline can be made compatible, then you can use this Snap by dropping it on the canvas and selecting the child Pipeline for the Pipeline property. You will also want to enable the Reuse property to preserve the existing execution semantics of Nested Pipelines. The advantages of using this Snap over Nested Pipelines are:

• Multiple executions can be started to process documents in parallel.
• The Pipeline to execute can be determined dynamically by an expression.
• The original document or binary data is attached to any output documents or binary data if reuse is not enabled.

ForEach

Converting a ForEach Snap to the Pipeline Execute Snap is pretty straightforward since they have a similar set of properties. You should be able to select the Pipeline you would like to run and populate the parameter listing. The advantages of using this Snap over the ForEach Snap are:

• Documents or binary data fed into the Pipeline Execute Snap can be sent to the child Pipeline execution through an unlinked input view in the child.
  Documents or binary data sent out of an unlinked output view in the child Pipeline execution is written out of the Pipeline Execute's output view.
• The execution label can be changed.
• The Pipeline to execute can be determined dynamically by an expression.
• Executing a Pipeline does not require communication with the cloud servers. 

Task Execute

Converting a Task Execute Snap to the Pipeline Execute Snap is also straightforward since the properties are similar. To start, you only need to select the Pipeline you would like to use, you no longer have to create a Triggered Task. If you set the Batch Size property in the Task Execute to one, then you will not want to enable the Reuse property. If the Batch Size was greater than one, then you should enable Reuse. The Pipeline parameters should be the same between the Snaps. The advantages of using this Snap over the Task Execute Snap are:

• A task does not need to be created.
• Multiple executions can be started to process documents in parallel.
• There is no limit on the number of documents that can be processed by a single execution (i.e. no batch size).
• The execution label can be changed.
• The Pipeline to execute can be determined dynamically by an expression.
• The original document will be attached to any output documents if reuse is not enabled.
• Executing a Pipeline does not require communication with the cloud servers. 

"Auto" Router

Converting a Pipeline that uses the Router Snap in "auto" mode can be done by moving the duplicated portions of the Pipeline into a new Pipeline and then calling that Pipeline using a Pipeline Execute. After refactoring the Pipeline, you can adjust the "Pool Size" of the Pipeline Execute Snap to control how many operations are done in parallel. The advantages of using this Snap over an "Auto" Router are:

• De-duplication of Snaps in the Pipeline.
• Adjusting the level of parallelism is trivially done by changing the Pool Size value.

Creating a Child Pipeline by Accessing Pipelines in the Pipeline Catalog in Designer

You can now browse the Pipeline Catalog for the target child Pipeline, and then select, drag and drop it in the Canvas. The SnapLogic Designer automatically adds the child Pipeline using a Pipeline Execute Snap.

Likewise, you can preview a child Pipeline by hovering over a Pipeline Execute Snap while the parent Pipeline is open on the Designer Canvas.

Returning Child Pipeline Output to the Parent Pipeline

A common use case for the Pipeline Execute Snap is to run a child Pipeline whose output is immediately returned to the parent Pipeline for further processing. You can achieve this return with the following Pipeline design for the child Pipeline.

In this example, the document in this child Pipeline is sent to the parent Pipeline through output1 of the Copy Snap. Any unconnected output view is returned to the parent Pipeline. You can use any Snap that completes execution in this way.

Examples

Run a Child Pipeline Multiple Times

The PE_Multiple_Executions project demonstrates how you can configure the Pipeline Execute Snap to execute a child Pipeline multiple times. The project contains the following Pipelines:

• PE_Multiple_Executions_Child: A simple child Pipeline that writes out a document with static string and the number of input documents received by the Snap.
• PE_Multiple_Executions_NoReuse_Parent:  A parent Pipeline that executes the PE_Multiple_Executions_Child Pipeline five times. You can save the Pipeline to examine the output documents. Note that the output contains a copy of the original document and the $inCount field is always set to one because the Pipeline was separately executed five times.
• PE_Multiple_Executions_Reuse_Parent: A parent Pipeline that executes the PE_Multiple_Executions_Child Pipeline once and feeds the child Pipeline execution five documents. You can save the Pipeline to examine the output documents. Note that the output does not contain a copy of the original document and the $inCount field goes up for each document since the same Snap instance is being used to process each document.
• PE_Multiple_Executions_UltraSplitAggregate_Parent: A parent Pipeline that is an example of using Snaps that are not Ultra-compatible in an Ultra Pipeline. This Pipeline can be turned into an Ultra Pipeline by removing the JSON Generator Snap at the head of the Pipeline and creating an Ultra Task.
• PE_Multiple_Executions_UltraSplitAggregate_Child: A child Pipeline that splits an array field in the input document and sums the values of the $num field in the resulting documents.

Propagate a Schema Backward – 1

The project, PE_Backward_Schema_Propagation_Contacts, demonstrates the schema suggest feature of the Pipeline Execute Snap. It contains the following files:

• PE_Backward_Schema_Propagation_Contacts_Parent
• PE_Backward_Schema_Propagation_Contacts_Child
• contact.schema (Schema file)
• test.json (Output file)

The parent Pipeline is shown below:

The child Pipeline is as shown below:

The Pipeline Execute Snap is configured as:

The following schema is provided in the JSON Formatter Snap. It has three properties - $firsName, $lastName, and $age. This schema is back propagated to the parent Pipeline.

The parent Pipeline must be validated in order for the child Pipeline's schema to be back-propagated to the parent Pipeline. Below is the Mapper Snap in the parent Pipeline: 

Notice that the Target Schema section shows the three properties of the schema in the child Pipeline:

Upon execution the data passed in the Mapper Snap will be written into the test.json file in the child Pipeline. The exported project is available in the Downloads section below.

Propagate Schema Backward and Forward

Image Modified

• The string expression $foo is propagated from the child Pipeline to the Pipeline Execute Snap.
• The Pipeline Execute Snap propagates it to the upstream Mapper Snap (Mapper_InputSchemaPropagation), as visible in the Target Schema section. Here it is assigned the value 123.
• This is passed from the Mapper to the Pipeline Execute Snap that internally passes the value to the child Pipeline. Here $foo is mapped to $bar. $baz is another string expression in the child Pipeline (assigned the value 2).
• $bar, and $baz are propagated to the Pipeline Execute Snap and propagated forward to the downstream Mapper Snap (Mapper_TargetSchemaPropagation). This can be seen in the Input Schema section of the Mapper Snap.