In this article
PipeLoop Snap
Demo slides: https://docs.google.com/presentation/d/1rTjUi9kRgcxrkQqkvsIOB0_C3jjahivTXNZ0qK9_Bcs/edit?usp=sharing
Overview
You can use this Snap to enable looping on pipeline executions (do while loop) based on a stopping condition or a hard cutoff iteration limit. The output document of the snap will be fed into the next round of document input for execution until the stop condition is met or the cutoff limit is reached.
Snap Type
The PipeLoop Snap is a Flow-type Snap.
Prerequisites
Valid client ID.
A valid account with the required permissions.
Support for Ultra Pipelines
Works in Ultra Pipelines if batching is disabled.
Limitations
Reuse is not currently supported because of needing to close an input view in order to force a flush of all documents out.
PipeLoop supports child pipelines with 1 and only 1 open input view and 1 and only 1 open output view.
PipeLoop will read only the first output document for a given fed-in input document (all others will be ignored and dropped).
We will be able to support expression on the pipeline specification, but it will only evaluate on the input document to PipeLoop, not for each iteration.
Known Issues
None.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input |
|
|
| The document or binary data to send to the child Pipeline. |
Output |
|
|
| |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
Snap Settings
Expression icon (): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
Upload icon (): Indicates that you can upload files. Learn more about managing Files.
Field Name | Field Type | Description | |
---|---|---|---|
Label* Default Value: ELT Database Account | String | The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | |
Pipeline* Default Value: N/A | String/Expression | Specify the path of the pipeline to execute. | |
Pipeline Parameters | Use this field set to define child pipeline parameters to use for executing the pipeline. | ||
Parameter name Default Value: <value> or None. | String/Suggestion | Specify the name of the parameter. You can select the Pipeline Parameters defined for the Pipeline selected in the Pipeline field. | |
Parameter value Default Value: <value> or None. | String/Expression | Specify the value for the pipeline parameter, which can be an expression based on incoming documents or a constant. If you configure the value as an expression based on the input, then each incoming document or binary data is evaluated against that expression when invoking the pipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next. | |
Loop options | Define the iteration configuration. | ||
Stop condition
| String/Expression | Specify the condition to be evaluated during execution. The loop execution stops when the stop condition is met. | |
Iteration limit
| Integer/Expression | Specify the maximum limit for the number of iterations to run. | |
Debug iteration outputs
| Checkbox | Select this checkbox to list the documents from each iteration, index and stop condition evaluation for debug purposes. | |
Execution options | Define the child pipeline execution configuration. | ||
Execute on
| Dropdown list | Select the one of the following Snaplex options to specify the target Snaplex for the child Pipeline:
| |
Snaplex path
| String/Expression | Appears when you choose the SNAPLEX_WITH_PATH option. Specify the Snaplex to execute on. | |
Execution label
| String/Expression | Specify the label to display in the Dashboard for the current execution. | |
Snap Execution Default Value: | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Troubleshooting
Error | Reason | Resolution |
---|---|---|
Account validation failed. | The Pipeline ended before the batch could complete execution due to a connection error. | Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open. |
Examples
Excluding Fields from the Input Data Stream
We can exclude the unrequired fields from the input data stream by omitting them in the Input schema fieldset. This example demonstrates how we can use the <Snap Name> to achieve this result:
<screenshot of Pipeline/Snap and description>
Download this Pipeline.
Downloads
Download and import the pipeline into the SnapLogic application.
Configure Snap accounts, as applicable.
Provide pipeline parameters, as applicable.