...

In this Article

...

Snap type:

...

Flow

...

Description:

This Snap is for executing Pipelines and passing documents into and out of these executions. The following execution modes are supported by this Snap:

• Execute a Pipeline for every input document. If the target Pipeline has an unlinked input view, then the input document to this Snap is passed into the unlinked input of the child execution. Similarly, if the target Pipeline has an unlinked output view, then the output document from the child Pipeline execution is used as the output document from the Pipeline Execute Snap.
• Execute one or more Pipelines and reuse them to process multiple input documents to this Snap. The child Pipeline must have one unlinked input view and one unlinked output view. As documents are received by this Snap, they are passed to the child Pipeline executions for processing. Any output documents from the child Pipeline executions are used as the output of this Snap.

You can execute Pipelines on the same Snaplex node that the parent Pipeline is running without contacting the SnapLogic cloud. In this mode, the Snap does not need to connect with the SnapLogic cloud servers, allowing you to use this Snap in an Ultra Pipeline. For example, you can use Snaps that do not support Ultra mode with this design.

This Snap will eventually replace the ForEach and Task Execute Snaps, as well as the Nested Pipeline mechanism.

...

None

...

...

Accounts are not used with this Snap.

...

The Input/Output views on the Views tab are configurable.

...

This Snap has at most one document or binary data output view. 

• If the child Pipeline has an unlinked output,  then documents or binary data from that view are passed out of this view.
• If the child Pipeline does not have an unlinked view, then output document or binary data is generated for successful runs with the following field: run_id.

Unsuccessful runs write a document to the error view. If reuse is disabled, then the original input document or binary data is added to output document.

...

Settings

...

In this article

Overview

The Pipeline Execute Snap executes a pipeline in a specific Snaplex with the specified parameters. You can use this Snap to execute multiple child pipelines from a single parent pipeline. Configure the Snap to execute the pipeline and data input.

Image Added

Key Features

The Pipeline Execute Snap enables you to:

• Structure complex pipelines into smaller segments through child pipelines

• Initiate parallel data processing using the pooling option

• Orchestrate data processing across nodes, in the Snaplex or across Snaplexes

• Distribute global values through pipeline parameters across a set of child pipeline Snaps

Supported Modes for Pipelines

Prerequisites

None.

Limitations

• If there are insufficient Snaplex nodes to execute the pipeline, the Snap waits until the resources become available. In this scenario, a message appears in the execution statistics dialog.

  Image Added

• Only the last 100 completed child pipeline runs are saved for inspection in the Dashboard because this Snap generates many pipeline runtimes.

• The Pipeline Execute Snap cannot exceed a depth of 64 child pipelines before they begin to fail.

• The child pipelines do not display data preview details. However, you can view the data preview for any child pipeline after the Pipeline Execute Snap completes execution in the parent pipeline.

• Ultra Pipelines do not support batching. 

• Unlike the Group By N Snap, when you configure the Batch field, the documents are processed one by one by the Pipeline Execute Snap and then transferred to the child pipeline when the parent pipeline receives it. The child pipeline closes when the batch or input stream ends.

Known Issues

None.

Snap Views

Snap Settings

...

...

...

...

...

...

...

...

...

...

...

...

...

The child Pipeline must be designed to be compatible with this Snap. The current requirements are:

No unlinked binary input or output views.

...

...

Select the Snaplex or node on which you want to execute the child Pipeline:

...

Default value: LOCAL_NODE

...

...

...

...

...

...

...

The label to display in the Pipeline view of the Dashboard. By default, the child Pipeline's label is used.  You can use this property to differentiate one execution from another.

Default value: N/A

...

You can configure Pipeline parameters used for executing the Pipeline in this property. Select the Pipeline parameters defined for the Pipeline selected in the Pipeline field. The Parameter value is the expression property that you can configure as an expression based on incoming documents, or this value can be a constant.

...

The Parameter name is the name of the parameter used while executing the Pipeline. You can select the Pipeline parameters defined for the Pipeline selected in the Pipeline field.

Default value: N/A

...

The Parameter value is the expression property that can be configured as an expression based on incoming documents or can be entered as a constant.

...

...

...

...

...

...

...

...

...

Ultra Mode Compatibility

 If reuse is enabled and this Snap is used in an Ultra Pipeline, then the Snaps in the child Pipeline must also be Ultra-compatible.

If you need to use Snaps that are not Ultra-compatible in an Ultra Pipeline, you can create a child Pipeline with those Snaps and use a Pipeline Execute Snap with reuse disabled to invoke the Pipeline. Since the child Pipeline is executed for every input document, the Ultra Pipeline restrictions do not apply. 

For example, if you want to run an SQL Select operation on a table that would return more than one document, you can put a Select Snap followed by a Group By N Snap with the group size set to zero in a child Pipeline. In that configuration, the child Pipeline is executed, performs the select operation, and then the Group By Snap gathers all of the outputs into a single document or binary data. That single output document or binary data can then be used as the output of the Ultra Pipeline.

Default value:  Not selected

...

The maximum number of retry attempts that the Snap must make in case of a network failure. If the child Pipeline does not execute successfully, an error document is written to the error view.

Example: 3

Default value: 0

...

The minimum number of seconds for which the Snap must wait between two successive retry requests. A retry happens only when the previous attempt resulted in an exception. 

Example:  10

Default value: 1

...

Multiple input documents or binary data can be processed concurrently by specifying an execution pool size. When the pool size is greater than one, the Snap starts executions as needed up to the given pool size. When Reuse is enabled, the Snap starts a new execution only if all executions are busy working on documents or binary data and the total number of executions is below the pool size. However, in Spark mode, if you execute multiple Pipelines simultaneously, the Hadoop cluster might face resource constraints, depending upon the complexity of the Pipelines and the size of the documents being processed. 

Default value: 1

...

The maximum number of seconds for which the Snap must wait for the child Pipeline to complete execution. If the child Pipeline does not complete executing before the timeout, the execution process stops and is marked as failed.

Example:  10

Default value: 0 (No timeout is the default)

...

...

Examples

Execute Child Pipeline Multiple Times

Converting Existing Pipelines

Propagate Schema Backward – 1

...

The project, PE_Backward_Schema_Propagation_Contacts, demonstrates the schema suggest feature of the Pipeline Execute Snap. It contains the following files:

• PE_Backward_Schema_Propagation_Contacts_Parent
• PE_Backward_Schema_Propagation_Contacts_Child
• contact.schema (Schema file)
• test.json (Output file)

The parent Pipeline is shown below:

Image Removed

The child Pipeline is as shown below:

Image Removed

The Pipeline Execute Snap is configured as:

Image Removed

The following schema is provided in the JSON Formatter Snap. It has three properties - $firsName, $lastName, and $age. This schema is back propagated to the parent Pipeline.

Image Removed

The parent Pipeline must be validated in order for the child Pipeline's schema to be back-propagated to the parent Pipeline. Below is the Mapper Snap in the parent Pipeline: 

Image Removed

Notice that the Target Schema section shows the three properties of the schema in the child Pipeline:

Image RemovedImage Removed

Upon execution the data passed in the Mapper Snap will be written into the test.json file in the child Pipeline. The exported project is available in the Downloads section below.

 Schema Propagation in Parent Pipeline – 2

Propagate Schema Backward and Forward

...

The project, PE_Backward_Forward_Schema_Propagation, demonstrates the Pipeline Execute Snap's capability of propagating schema in both directions – upstream as well as downstream. It contains the following Pipelines:

• PE_Backward_Forward_Schema_Propagation_Parent
• PE_Backward_Forward_Schema_Propagation_Child

Image Removed

Image Removed

Image Removed

 Image Removed

• The string expression $foo is propagated from the child Pipeline to the Pipeline Execute Snap.
• The Pipeline Execute Snap propagates it to the upstream Mapper Snap (Mapper_InputSchemaPropagation), as visible in the Target Schema section. Here it is assigned the value 123.
• This is passed from the Mapper to the Pipeline Execute Snap that internally passes the value to the child Pipeline. Here $foo is mapped to $bar. $baz is another string expression in the child Pipeline (assigned the value 2).
• $bar, and $baz are propagated to the Pipeline Execute Snap and propagated forward to the downstream Mapper Snap (Mapper_TargetSchemaPropagation). This can be seen in the Input Schema section of the Mapper Snap.

The exported project is available in the Downloads section below.

Downloads

...

Pipeline parameter values can only be changed if this flag is not enabled. In other words, reusable executions cannot have different Pipeline parameter values for different documents.

Troubleshooting

Access pipelines in the Pipeline Catalog in the Designer to create a Child pipeline

You can now browse the pipeline catalog for the target child pipeline and then select, drag, and drop it into Canvas. The Designer automatically adds the child pipeline using the Pipeline Execute Snap. You can preview a child pipeline by hovering over a Pipeline Execute Snap while the parent pipeline is open on the Designer.

Return child pipeline output to the parent pipeline

A common use case for the Pipeline Execute Snap is to run a child Pipeline whose output is immediately returned to the parent pipeline for further processing. You can achieve this return with the following Pipeline design for the child Pipeline.

In this example, the document in this child Pipeline is sent to the parent Pipeline through output1 of the child pipeline. Any unconnected output view is returned to the parent Pipeline. You can use any Snap that completes execution in this way.

...

Execution States

When a Pipeline Execute Snap activates its child pipeline, you can view the status as it executes on the SnapLogic Designer canvas.

A child pipeline also reports its execution to the parent pipeline. In the Monitor overview and Dashboard, you can hover over the status shown in the Status column for a pipeline with a Pipeline Execute Snap, and the following messages are displayed for the following scenarios:

• If the parent Pipeline has the status Failed shown in the Status column, the following message is displayed: One of the child Pipelines Failed.

• If the parent Pipeline has the status Completed with Errors shown in the Status column, the following message is displayed: One of the child Pipelines completed with Errors.

These execution state messages apply even when the child pipeline does not appear on the Dashboard because of child pipeline execution limits.

The Monitor Execution overview does not include the Completed with Warnings status as a searchable status.

Examples

Run a Child pipeline multiple times

The PE_Multiple_Executions project demonstrates how to configure the Pipeline Execute Snap to execute a child pipeline multiple times. The project contains the following pipelines:

• PE_Multiple_Executions_Child: This is a simple child pipeline that writes out a document with a static string and the number of input documents received by Snap.

• PE_Multiple_Executions_NoReuse_Parent: This parent pipeline executes the PE_Multiple_Executions_Child Pipeline five times. You can save the Pipeline to examine the output documents. Note that the output contains a copy of the original document, and the $inCount field is always set to one because the Pipeline was separately executed five times.

• PE_Multiple_Executions_Reuse_Parent: This parent Pipeline executes the PE_Multiple_Executions_Child Pipeline once and feeds the child Pipeline execution five documents. You can save the Pipeline to examine the output documents. Note that the output does not contain a copy of the original document, and the $inCount field goes up for each document because the same Snap instance is being used to process each document.

• PE_Multiple_Executions_UltraSplitAggregate_Parent: This parent Pipeline is an example of using Snaps that are not Ultra-compatible in an Ultra Pipeline. It can be turned into an Ultra Pipeline by removing the JSON Generator Snap at the head of the Pipeline and creating an Ultra Task.

• PE_Multiple_Executions_UltraSplitAggregate_Child: A child Pipeline that splits an array field in the input document and sums the values of the $num field in the resulting documents.

Propagate a Schema Backward 

The project, PE_Backward_Schema_Propagation_Contacts, demonstrates the schema that suggests a feature of the Pipeline Execute Snap. It contains the following files:

• PE_Backward_Schema_Propagation_Contacts_Parent

• PE_Backward_Schema_Propagation_Contacts_Child

• contact.schema (Schema file)

• test.json (Output file)

The parent pipeline is shown below:

...

The child Pipeline is as shown below:

...

The Pipeline Execute Snap is configured as:

The following schema is provided in the JSON Formatter Snap. It has three properties - $firsName, $lastName, and $age. This schema is back propagated to the parent pipeline.

...

The parent Pipeline must be validated in order for the child Pipeline's schema to be back-propagated to the parent Pipeline. Below is the Mapper Snap in the parent pipeline: 

...

Notice that the Target Schema section shows the three properties of the schema in the child pipeline:

...

On execution, the data passed in the Mapper Snap will be written into the test.json file in the child Pipeline. The exported project is available in the Downloads section below.

Propagate Schema Backward and Forward

The project, PE_Backward_Forward_Schema_Propagation, demonstrates the Pipeline Execute Snap's capability of propagating schema in both directions – upstream and downstream. It contains the following Pipelines:

• PE_Backward_Forward_Schema_Propagation_Parent

• PE_Backward_Forward_Schema_Propagation_Child

The parent Pipeline is as shown below:

...

The Pipeline Execute Snap is configured to call the Pipeline schema-child. This child Pipeline consists of a Mapper Snap that is configured as shown below:

...

The Mapper Snaps upstream and downstream of the Pipeline Execute Snap: Mapper_InputSchemaPropagation and  Mapper_TargetSchemaPropagation are configured as shown below:

...

...

When the Pipeline is executed, data propagation takes place between the parent and child Pipeline:

• The string expression $foo is propagated from the child Pipeline to the Pipeline Execute Snap.

• The Pipeline Execute Snap propagates it to the upstream Mapper Snap (Mapper_InputSchemaPropagation), as visible in the Target Schema section. Here it is assigned the value 123.

• This is passed from the Mapper to the Pipeline Execute Snap, which internally passes the value to the child Pipeline. Here, $foo is mapped to $bar. $baz is another string expression in the child Pipeline (assigned the value 2).

• $bar, and $baz are propagated to the Pipeline Execute Snap and propagated forward to the downstream Mapper Snap (Mapper_TargetSchemaPropagation). This can be seen in the Input Schema section of the Mapper Snap.

Migrating from Legacy-Nested pipelines

The Pipeline Execute Snap can replace some uses of the Nested Pipeline mechanism and the ForEach and Task Execute Snaps. For now, this Snap only supports child Pipelines with unlinked document views (binary views are not supported). If these limitations are not a problem for your use case, read on to find out how to transition to this Snap and the advantages of doing so.

Nested Pipeline

Converting a Nested Pipeline will require the child pipeline to be adapted to have no more than a single unlinked document input view and no more than a single unlinked document output view. If the child pipeline can be made compatible, then you can use this Snap by dropping it on the canvas and selecting the child pipeline for the Pipeline property.

You will also want to enable the Reuse property to preserve the existing execution semantics of Nested pipelines. The advantages of using this Snap over Nested pipelines are:

• Multiple executions can be started to process documents in parallel.

• The Pipeline to execute can be determined dynamically by an expression.

• The original document or binary data is attached to any output documents or binary data if reuse is not enabled.

ForEach

Converting the ForEach Snap to the Pipeline Execute Snap is a simple process. Select the pipeline to run and populate the parameters. The benefits of utilizing the Snap compared to the ForEach Snap are:

• Documents or binary data fed into the Pipeline Execute Snap can be sent to the child Pipeline execution through an unlinked input view in the child.
  Documents or binary data sent from an unlinked output view in the child Pipeline execution is written out of the Pipeline Execute's output view.

• The execution label can be changed.

• The Pipeline to execute can be determined dynamically by an expression.

• Executing a Pipeline does not require communication with the cloud servers. 

Task Execute

Converting a Task Execute Snap to a Pipeline Execute Snap is also straightforward because the properties are similar. 

To start, you only need to select the Pipeline you want to use; you no longer have to create a Triggered Task. If you set the Batch Size property in the Task Execute to one, then you will not want to enable the Reuse property. If the Batch Size was greater than one, then you should enable Reuse. The Pipeline parameters should be the same between the Snaps. The advantages of using this Snap to the Task Execute Snap are:

• A task does not need to be created.

• You can start multiple executions simultaneously to process documents in parallel.

• You can process an unlimited number of documents that can be processed with a single execution (that is, no batch size).

• The execution label can be changed.

• Determine the pipeline to execute dynamically by an expression.

• The original document is attached to any output documents if reuse is not enabled.

• No requirement to communicate with the cloud servers to execute a pipeline.

Converting a pipeline that uses the Router Snap in "auto" mode can be done by moving the duplicated portions of the Pipeline into a new pipeline and then calling that Pipeline using a Pipeline Execute. After refactoring the Pipeline, you can adjust the "Pool Size" of the Pipeline Execute Snap to control how many operations are done in parallel. The advantages of using this Snap over an "Auto" Router are:

• De-duplication of Snaps in the pipeline.

• Adjustment of the level of parallelism is trivial simply change the Pool Size value.

Downloads