Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

Table of Contents
minLevel1
maxLevel2
absoluteUrltrue

PipeLoop Snap

Demo slides: https://docs.google.com/presentation/d/1rTjUi9kRgcxrkQqkvsIOB0_C3jjahivTXNZ0qK9_Bcs/edit?usp=sharing

Properties

Pipeline

Pipeline Parameters

  • The pipeline parameters to pass into the child pipeline

  • Contains two child fields, Parameter Name and Parameter Value

  • Parameter Name

    • Suggestable - suggests from child pipeline pipeline parameters field key

    • String field

  • Parameter Value

    • Expression-enabled

    • Schema-enabled

    • String field

Loop options

Stop condition

  • Condition to be evaluated during execution. The loop execution stops when the stop condition is met

  • String field

  • expression-enabled

  • schema-enabled

Iteration limit

  • Hard cutoff to use to stop the pipeline iteration

  • Integer field

  • expression-enabled

  • required

Debug iteration outputs

  • List out iteration documents, loop index, and current evaluated stop condition

  • Boolean field

Execution options

Execute On

  • The environment to execute on

  • required

  • available values are

    • LOCAL_NODE, LOCAL_SNAPLEX, SNAPLEX_WITH_PATH

Snaplex Path

  • The Snaplex to execute on

  • String field

  • expression-enabled

  • Enabled when Execute On is set to SNAPLEX_WITH_PATH

PipeLoop Current Limitations

Table of Contents
stylenone

Overview

You can use this Snap to enable looping on pipeline executions (do while loop) based on a stopping stop condition or a hard cutoff iteration limit. The output document of the snap will be each iteration is fed into the next round of document input for execution iteration as the input document, and execution continues until the stop condition is met or the cutoff iteration limit is reached. Learn more.<Image>

...

Snap Type

The PipeLoop Snap is a Flow-type Snap.

Prerequisites

  • Valid client ID.

  • A valid account with the required permissionsChild pipelines must have one input view and one output view.

Support for Ultra

...

Tasks

Limitations

  • Reuse is currently not currently supported due to needing supported because of the requirement to close an input view in order to force a flush of all documents out

  • PipeLoop supports child pipelines with 1 and only 1 open input view and 1 and only 1 open output view

  • PipeLoop will read only .

  • Reads only the first output document for a given fed-in specific input document (all others will be the remaining documents are ignored and dropped)We will be able to support expression .

  • Supports expressions on the pipeline specification, but it will only evaluate on evaluates the input document to PipeLoop, not for each iteration.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

  • Binary

  • Document

  • Min: 1

  • Max: 1

  • Mapper

  • Copy

The document or binary data to send to the child

Pipeline

pipeline. For a single document input, only the first output document is captured in the output document and the remaining documents are ignored.

Output

  • Binary

  • Document

 

  • Min: 1

  • Max: 1

  • Mapper

If you select the Debug iterations output checkbox, an additional _iteration_documents is added to the output document alongside the output documents and passthrough documents.

_iteration_documents contains the following fields:

  • iteration: The index of the loop iteration.

  • child_pipeline_output: The output document of the respective iteration.

  • stop_condition_evaluation: The evaluated result of the stop condition.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in

Pipelines

pipelines.

Snap Settings

Info
  • Expression icon ((blue star)): Indicates the value is an expression : JavaScript syntax to access SnapLogic Expressions to set field values dynamically (if enabled) or . If disabled, you can provide a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • SnapGPT (blue star): Generates SnapLogic Expressions based on natural language using SnapGPT. Learn more.

  • Suggestion icon ((blue star)): Indicates Populates a list that is of values dynamically populated based on the your Account configuration.

  • Upload icon ((blue star)): Indicates that you can upload Uploads files. Learn more about managing Files.

Learn more about the icons in the Snap Settings dialog.

Field Name

Field Type

Description

Label*

 

Default Value

ELT Database Account

PipeLoop
Example

ELT RS Account

SimpleLoop

String

The

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same

Snap

Snaps in your pipeline.

 

Pipeline*

 

Default Value: N/A
Example: demo-child-pipeline

String/Expression

Specify the path of the pipeline to execute.

Pipeline Parameters

Use this field set to define child pipeline parameters to use for executing the pipeline.

Parameter name

Default Value:

 <value> or None.

 N/A
Example

<value>

tools

String/Suggestion

Specify

the name of

the parameter's name. You can also select the Pipeline Parameters defined for the

Pipeline

pipeline selected in the Pipeline field.

Parameter value

Default Value:

 <value> or None.

 N/A
Example

<value>

$tools

String/Expression

Specify the value for the

Pipeline Parameter

pipeline parameter, which can be an expression based on incoming documents or a constant. 

If you configure the value as an expression based on the input, then each incoming document or binary data is evaluated against that expression when invoking the

Pipeline

pipeline. The result of the expression is JSON-encoded if it is not a string. The child

Pipeline

pipeline then needs to use the JSON.parse() expression to decode the parameter value.

When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next.

Loop options

Define the iteration configuration.

Stop condition

Default Value:

 

 N/A
Example: $num >= 10

String/Expression

Specify the condition to be evaluated during execution that decides whether the execution should stop or continue. The iteration continues until the stop condition is met or the iteration limit is reached. The Snap errors out if the Stop condition cannot be evaluated at run time. The stop condition is evaluated on the output document from each iteration.

Iteration limit

Default Value10
Example: 15

Integer/Expression

Specify the maximum limit for the number of iterations to run to avoid infinite loops.

Debug iteration outputs

Default Value


Example:Checkbox

Deselected

Checkbox

Select this checkbox to include the index, output document, and stop condition evaluation for each iteration as part of the output format.

 

Execution options

Define the child pipeline execution configuration.

Execute on

Default ValueLOCAL_NODE
Example: LOCAL_SNAPLEX

Dropdown list

Select

the

one of the following Snaplex options to specify the target Snaplex for the child

Pipeline:
  • SNAPLEX_WITH_PATH. Runs the child Pipeline on a user-specified Snaplex. When you choose this option, the Snaplex Path field appears.

  • pipeline:

    • LOCAL_NODE

    .
    • : Runs the child

    Pipeline
    • pipeline on the same node as the parent

    Pipeline
    • pipeline.

    • LOCAL_SNAPLEX

    . Runs
    • : Runs the child

    Pipeline
    • pipeline on one of the available nodes in the same Snaplex as the parent

    Pipeline
    • pipeline.

    Execution label

    Default Value: N/A
    Example: demo

    String/Expression

    Specify the label to display in the Dashboard for the current execution.

    Snap Execution

    Default ValueValidate & Execute
    Example:

    Validate &

    Execute only

    Dropdown list

    Select one of the following three modes in which the Snap executes:

    • Validate & Execute: Performs limited execution of the Snap and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

    • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

    • Disabled: Disables the Snap and all Snaps that are downstream from it.

    ...

    Example

    ...

    Error

    ...

    Reason

    ...

    Resolution

    ...

    Account validation failed.

    ...

    The Pipeline ended before the batch could complete execution due to a connection error.

    ...

    Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open.

    Examples

    Excluding Fields from the Input Data Stream

    We can exclude the unrequired fields from the input data stream by omitting them in the Input schema fieldset. This example demonstrates how we can use the <Snap Name> to achieve this result:

    <screenshot of Pipeline/Snap and description>

    Download this Pipeline. 

    Downloads

    Info
    1. Download and import the pipeline into the SnapLogic application.

    2. Configure Snap accounts, as applicable.

    3. Provide pipeline parameters, as applicable.

    Attachments
    patterns*.slp, *.zip
    sortByname

    Snap Pack History

    ...

    titleClick here to expand...

    ...

    Related Content

    ...

    Endpoint Doc Link 1

    ...

    Endpoint Doc Link 2

    ...

    Getting Started with SnapLogic

    ...

    Insert excerpt
    Flow Snap Pack
    Flow Snap Pack
    nopaneltrue
    Related links: