Skip to end of banner
Go to start of banner

WIP-PipeLoop

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

In this article

Overview

You can use this Snap to enable looping on pipeline executions based on a stopping condition or iteration limit. The output document of the Snap will be fed into the next round of document input for execution until the stop condition is met or the cutoff limit is reached.

pipe-loop-snap-overview.png

Snap Type

The PipeLoop Snap is a Flow-type Snap.

Prerequisites

None.

Support for Ultra Pipelines  

Limitations

  • Reuse is currently not supported because of needing to close an input view in order to force a flush of all documents out.

  • Supports child pipelines with one open input view and one open output view.

  • Reads only the first output document for a given input document (the rest will be ignored and dropped).

  • Supports expressions on the pipeline specification, but it will only be evaluated on the input document to PipeLoop, not for each iteration.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

  • Binary

  • Document

  • Min: 1

  • Max: 1

  • Mapper

  • Copy

The document or binary data to send to the child pipeline. For a single document input, only the first output document is captured in the output document.

Output

  • Binary

  • Document

  • Min: 1

  • Max: 1

  • Mapper

If you select the Debug iterations output checkbox, an additional _iteration_documents is added to the output document along side the output documents and passthrough documents.

_iteration_doucuments contains the following fields:

  • iteration - the index of the loop iteration.

  • child_pipeline_output - the output document of the respective iteration.

  • stop_condition_evaluation - the evaluated result of the stop condition.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Expression icon ((blue star)): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Upload icon ((blue star)): Indicates that you can upload files. Learn more about managing Files.

Field Name

Field Type

Description

Label*

Default ValuePipeLoop
ExampleSimpleLoop

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Pipeline*

Default Value: N/A
Example: demo-child-pipeline

String/Expression

Specify the path of the pipeline to execute.

Pipeline Parameters

Use this field set to define child pipeline parameters to use for executing the pipeline.

Parameter name

Default Value: N/A
Exampletools

String/Suggestion

Specify the name of the parameter. You can select the Pipeline Parameters defined for the Pipeline selected in the Pipeline field.

Parameter value

Default Value: N/A
Example$tools

String/Expression

Specify the value for the pipeline parameter, which can be an expression based on incoming documents or a constant. 

If you configure the value as an expression based on the input, then each incoming document or binary data is evaluated against that expression when invoking the pipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the JSON.parse() expression to decode the parameter value.

When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next.

Loop options

Define the iteration configuration.

Stop condition


Default Value: N/A
Example: $num >= 10

String/Expression

Specify the condition to be evaluated during execution that decides whether the execution should stop or continue.

  • The loop execution stops when the stop condition is met.

  • The Snap errors out if the Stop condition cannot be evaluated at run time.

Iteration limit


Default Value10
Example: 15

Integer/Expression

Specify the maximum limit for the number of iterations to run to avoid infinite loops.

Debug iteration outputs


Default Value
Example:

Checkbox

Select this checkbox to list the documents from each iteration, index and stop condition evaluation for debug purposes.

Execution options

Define the child pipeline execution configuration.

Execute on


Default ValueLOCAL_NODE
Example: SNAPLEX_WITH_PATH

Dropdown list

Select the one of the following Snaplex options to specify the target Snaplex for the child Pipeline:

  • SNAPLEX_WITH_PATH: Runs the child pipeline on a user-specified Snaplex.

  • LOCAL_NODE: Runs the child pipeline on the same node as the parent pipeline.

  • LOCAL_SNAPLEX: Runs the child pipeline on one of the available nodes in the same Snaplex as the parent pipeline.

Snaplex path


Default Value: N/A
Example:

String/Expression

Appears when you choose the SNAPLEX_WITH_PATH option.

Specify the Snaplex to execute on.

Execution label


Default Value: N/A
Example: demo

String/Expression

Specify the label to display in the Dashboard for the current execution.

Snap Execution

Default Value
Example: Validate & Execute

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Execution flow of the PipeLoop Snap

The Snap follows the following procedure:

  1. Passes input document to the child pipeline.

  2. Executes the child pipeline.

  3. Collects child pipeline output document.

  4. Evaluates the stop condition based on the child pipeline output document.

  5. If the stop condition is false, the Snap checks if the iteration limit is reached.

  6. If the iteration limit is not reached, the Snap passes the output document to input (back to Step 1). The iteration stops if the stop condition is evaluated to true or the iteration limit is reached.

Troubleshooting

Error

Reason

Resolution

Account validation failed.

The Pipeline ended before the batch could complete execution due to a connection error.

Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open.

Examples

Excluding Fields from the Input Data Stream

We can exclude the unrequired fields from the input data stream by omitting them in the Input schema fieldset. This example demonstrates how we can use the <Snap Name> to achieve this result:

<screenshot of Pipeline/Snap and description>

Download this Pipeline. 

Downloads

  1. Download and import the pipeline into the SnapLogic application.

  2. Configure Snap accounts, as applicable.

  3. Provide pipeline parameters, as applicable.

  File Modified
No files shared here yet.

Snap Pack History

 Click here to expand...
Error rendering macro 'excerpt-include' : No link could be created for 'XYZ Snap Pack'.

Related Content

  • No labels