Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

Table of Contents
minLevel1
maxLevel2
absoluteUrltrue

...

Demo slides: https://docs.google.com/presentation/d/1rTjUi9kRgcxrkQqkvsIOB0_C3jjahivTXNZ0qK9_Bcs/edit?usp=sharing

Properties

Pipeline

Pipeline Parameters

...

...

Contains two child fields, Parameter Name and Parameter Value

...

Parameter Name

  • Suggestable - suggests from child pipeline pipeline parameters field key

  • String field

...

Parameter Value

  • Expression-enabled

  • Schema-enabled

  • String field

Loop options

Stop condition

  • Condition to be evaluated during execution. The loop execution stops when the stop condition is met

  • String field

  • expression-enabled

  • schema-enabled

Iteration limit

  • Hard cutoff to use to stop the pipeline iteration

  • Integer field

  • expression-enabled

  • required

Debug iteration outputs

  • List out iteration documents, loop index, and current evaluated stop condition

  • Boolean field

Execution options

Execute On

  • The environment to execute on

  • required

  • available values are

    • LOCAL_NODE, LOCAL_SNAPLEX, SNAPLEX_WITH_PATH

Snaplex Path

  • The Snaplex to execute on

  • String field

  • expression-enabled

  • Enabled when Execute On is set to SNAPLEX_WITH_PATH

PipeLoop Current Limitations

Overview

You can use this Snap to enable looping on pipeline executions (do while loop) based on a stopping condition or a hard cutoff iteration limit. The output document of the snap will be fed into the next round of document input for execution until the stop condition is met or the cutoff limit is reached.<Image>

...

Snap Type

The PipeLoop Snap is a Flow-type Snap.

...

  • Reuse is not currently supported due to because of needing to close an input view in order to force a flush of all documents out.

  • PipeLoop supports child pipelines with 1 and only 1 open input view and 1 and only 1 open output view.

  • PipeLoop will read only the first output document for a given fed-in input document (all others will be ignored and dropped).

  • We will be able to support expression on the pipeline specification, but it will only evaluate on the input document to PipeLoop, not for each iteration.

Known Issues

None.

Snap Views

...

Field Name

Field Type

Description

Label*

Default ValueELT Database Account
ExampleELT RS Account

String

The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Pipeline*

Default Value: N/A
Example: demo-child-pipeline

String/Expression

Specify the path of the pipeline to execute.

Pipeline Parameters

Use this field set to define child pipeline parameters to use for executing the pipeline.

Parameter name

Default Value<value> or None.
Example<value>

String/Suggestion

Specify the name of the parameter. You can select the Pipeline Parameters defined for the Pipeline selected in the Pipeline field.

Parameter value

Default Value<value> or None.
Example<value>

String/Expression

Specify the value for the Pipeline Parameterpipeline parameter, which can be an expression based on incoming documents or a constant. 

If you configure the value as an expression based on the input, then each incoming document or binary data is evaluated against that expression when invoking the Pipelinepipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the JSON.parse() expression to decode the parameter value.

When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next.

Loop options

Define the iteration configuration.

Stop condition


Default Value:  N/A
Example:

String/Expression

Specify the condition to be evaluated during execution. The loop execution stops when the stop condition is met.

Iteration limit


Default Value10
Example: 15

Integer/Expression

Specify the maximum limit for the number of iterations to run.

Debug iteration outputs


Default Value
Example:

Checkbox

Select this checkbox to list the documents from each iteration, index and stop condition evaluation for debug purposes.

Execution options

Define the child pipeline execution configuration.

Execute on


Default ValueLOCAL_NODE
Example: SNAPLEX_WITH_PATH

Dropdown list

Select the one of the following Snaplex options to specify the target Snaplex for the child Pipeline:

  • SNAPLEX_WITH_PATH.: Runs the child Pipeline on a user-specified Snaplex. When you choose this option, the Snaplex Path field appears.

  • LOCAL_NODE.: Runs the child Pipeline on the same node as the parent Pipeline.

  • LOCAL_SNAPLEX. Runs : Runs the child Pipeline on one of the available nodes in the same Snaplex as the parent Pipeline.

Snaplex path


Default Value: N/A
Example:

String/Expression

Appears when you choose the SNAPLEX_WITH_PATH option.

Specify the Snaplex to execute on.

Execution label


Default Value: N/A
Example: demo

String/Expression

Specify the label to display in the Dashboard for the current execution.

Snap Execution

Default Value
Example: Validate & Execute

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Troubleshooting

Error

Reason

Resolution

Account validation failed.

The Pipeline ended before the batch could complete execution due to a connection error.

Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open.

Examples

Excluding Fields from the Input Data Stream

...

Expand
titleClick here to expand...
Insert excerpt
XYZ Snap Pack
XYZ Snap Pack

...

Related Content

...