Skip to end of banner
Go to start of banner

Pipeline Execute

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 131 Current »

In this article

Overview

The Pipeline Execute Snap executes a pipeline in a specific Snaplex with the specified parameters. You can use this Snap to execute multiple child pipelines from a single parent pipeline. Configure the Snap to execute the pipeline and data input.

To execute a SnapLogic pipeline that is exposed as a REST service, use the HTTP Client Snap.

image-20240802-125822.png

Key Features

The Pipeline Execute Snap enables you to:

  • Structure complex pipelines into smaller segments through child pipelines

  • Initiate parallel data processing using the pooling option

  • Orchestrate data processing across nodes, in the Snaplex or across Snaplexes

  • Distribute global values through pipeline parameters across a set of child pipeline Snaps

Supported Modes for Pipelines

Modes

Description

Standard mode (default)

  • A new child pipeline is launched for each input document in the standard mode.

  • If you set the pool size to n (where n is any number with the default setting value of 1, then n concurrent child pipelines can run simultaneously. Each child pipeline processes one document from the parent and then completes.

Reuse mode

  • In reuse mode, child pipelines are started, and each child pipeline instance can process multiple input documents from the parent.

  • If you set the pool size to n (default 1), then n number of child pipelines are started and they will process the input document in a streaming manner. The child pipeline needs to have an unlinked input view for use in reuse mode.

  • Reuse mode is more performant, but it has the restriction that the child pipeline has to be a streaming pipeline.

  • In reuse mode if a child pipeline fails, it causes the parent pipeline to fail.

Resumable Child pipeline

  • The resumable pipeline does not support the Pipeline Execute Snap.

  • A regular mode pipeline can use the Pipeline Execute Snap to call a Resumable mode pipeline. If the child pipeline is a Resumable pipeline, then the batch size cannot be greater than one. 

Ultra Task pipelines

  • Works in Ultra Task pipelines with the following exceptions:

    • The reuse option works in the Ultra pipeline if the child pipelines run on the same node. Running on a different node in the same Snaplex fails with a lineage error. For more information, refer to Snap Support for Ultra Pipelines.

    • Without reuse enabled, the one-in-one-out requirement for Ultra means batching is not supported. A runtime check will fail the parent pipeline if the batch size is set to greater than 1. This would be similar to the current behavior in DB insert and other snaps in ultra mode.

ELT Mode

  • A pipeline can only use the Pipeline Execute Snap to call another pipeline with ELT Snaps in Standard Mode.

Pooling Enabled

  • The pool size and batch size can both be set to greater than one, in which case the input documents are spread across the child pipeline in a round-robin method to ensure that if the child pipeline does any slow processing external calls, then the processing is spread across the children in parallel. The limitation of this option is that the document order is not maintained.

Replaces Deprecated Snaps: This Snap replaces the ForEach and Task Execute Snaps and the Nested pipeline mechanism.

Prerequisites

None.

Limitations

  • If there are insufficient Snaplex nodes to execute the pipeline, the Snap waits until the resources become available. In this scenario, a message appears in the execution statistics dialog.

  • Only the last 100 completed child pipeline runs are saved for inspection in the Dashboard because this Snap generates many pipeline runtimes.

  • The Pipeline Execute Snap cannot exceed a depth of 64 child pipelines before they begin to fail.

  • The child pipelines do not display data preview details. However, you can view the data preview for any child pipeline after the Pipeline Execute Snap completes execution in the parent pipeline.

  • Ultra Pipelines do not support batching. 

  • Unlike the Group By N Snap, when you configure the Batch field, the documents are processed one by one by the Pipeline Execute Snap and then transferred to the child pipeline when the parent pipeline receives it. The child pipeline closes when the batch or input stream ends.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Binary or Document

  • Min: 0

  • Max: 1

  • Mapper Snap

  • Copy Snap

The document or binary data to send to the child pipeline.

Retry is not supported if the input view is a Binary data type

Output

Binary or Document

  • Min: 0

  • Max: 1

  • Mapper Snap

  • Router Snap

  • If the child pipeline has an unlinked output, then documents or binary data from that view are transferred out of this view.

  • If the child pipeline has a linked view then the output document or binary data is generated for successful runtimes with the run_id field.

  • Unsuccessful pipeline runs write a document to the error view. If Reuse executions to process documents is deselected, the original input or binary data is added to the output document.

  • For Pool Size Usage: If the Pool Size field value is greater than one, then the order of records in the output is not guaranteed to be the same as that of the input.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current Pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the fieldset.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the fieldset.

  • Upload icon ((blue star) ): Indicates that you can upload files.

Field Name

Field Type

Field Dependency

Description

Label*

Default ValuePipeline Execute
ExampleExecuteCustomerUpdatePipeline

String

None.

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Pipeline*

Default Value: N/A
Example: 

String/Expression

None

Specify the child pipeline's absolute or relative (expression-based) path to run. If you specify only the pipeline name, Snap searches for the pipeline in the following folders in this order:

  • Current project

  • Shared project-space

  • Shared folder of the Org

You can specify the absolute path to the target project using the Org/project_space/project notation.

You can also dynamically choose the pipeline to run by entering an expression in this field when Expressions are enabled. For example, to run all of the pipelines in a project, you can connect the SnapLogic List Snap to this Snap to retrieve the list of pipelines in the project and run each one.

Execute On

Default Value: SNAPLEX_WITH_PATH
Example: groundplex4-West

Dropdown list

N/A

Select one of the following Snaplex options to specify the target Snaplex for the child pipeline:

  • LOCAL_NODE. Runs the child pipeline on the same node as the parent pipeline. This option is recommended when the child Pipeline is being used for Pipeline structuring and reuse rather than Pipeline workload distribution. Use this option for most regular child Pipeline executions.

  • LOCAL_SNAPLEX. Runs the child pipeline on one of the available nodes in the same Snaplex as the parent pipeline. As the child pipeline is on the local Snaplex, network communication is optimized for streaming data processing. Use this option only when workload distribution within the Snaplex is required.

  • SNAPLEX_WITH_PATH. Runs the child pipeline on a user-specified Snaplex, and the Snaplex Path field appears. It's important to note that this process is highly dependent on the network. The network communication is optimized for batch data processing since the child Pipeline is on a remote Snaplex. It is recommended to use this option only when the child Pipeline needs to run on a different Snaplex because of endpoint connectivity restrictions or for workload distribution purposes.

For more information, refer to the Best Practices.

Snaplex Path

Default Value:N/A
ExampleDevPlex-1

String/Expression

Appears when you select SNAPLEX_WITH_PATH for Execute On.

Enter the name of the Snaplex on which you want the child pipeline to run. Click (blue star)  to select from the list of Snaplex instances available in your Org.

Execution Label

Default Value: N/A 
Example: NetSuite-Create-Credit-Memo

String/Expression

N/A

Specify the label to display in the Pipeline view of the Dashboard.

You can use this field to differentiate each pipeline execution.

Pipeline Parameters

Use this fieldset to define the Pipeline Parameters for the pipeline selected in the Pipeline field. When you select Reuse executions to process documents, you cannot change parameter values from one Pipeline invocation to the next.

Parameter Name

Default ValueN/A
ExamplePostal_code

String

Debug mode checkbox is not selected.

Enter the name of the parameter. Select the defined Pipeline Parameters in the Pipeline field.

Parameter Value

Default Value: N/A
Example94002

String

None

Enter the value for the Pipeline Parameter, which can be an expression based on incoming documents or a constant. 

If you configure the value as an expression based on the input, then each incoming document or binary data evaluates against that expression when to you invoke the pipeline. The result of the expression is JSON-encoded if it is not a string. The child Pipeline then needs to use the JSON.parse() expression to decode the parameter value.

When Reuse executions to process documents is enabled, the parameter values cannot change from one invocation to the next.

Reuse executions to process documents

Default Value: Deselected

Checkbox

None

Select this checkbox to start a child pipeline and pass multiple inputs to the pipeline. Reusable executions continue to live until all of the input documents to this Snap are fully processed.

  • When you select this checkbox, and the pipeline parameters use expressions, the expressions are evaluated with the first document. The parameter value in the child pipeline does not change across documents.

  • If you do clear this checkbox, a new pipeline execution is created for each input document.  

  • Reuse mode does not support batching.

Batch size*

Default Value: 1
Example: 2

Integer/Expression

None

Specify the number of documents in the batch size. If Batch Size is set to N, then N input documents are sent to each child pipeline that is started. After N documents, the child pipeline input view is closed until the child pipeline completes its execution. The output of the child pipeline (one or more documents) passes to the Pipeline Execute output view. New child pipelines are started after the original pipeline is complete.

  • Batching is not supported for reuse executions..

  • This option is does not support reusable executions.

Pool Size*

Default Value: 1
Example: 4

Integer/Expression

None

Specify an execution pool size to process multiple input documents or binary data concurrently. When the pool size is greater than one, the Snap starts Pipeline executions as needed up to the specified pool size. 

When you select Reuse executions to process documents, the Snap starts a new execution only if either all executions are busy working on documents or binary data and the total number of executions is below the pool size.

Timeout (in seconds)

Default Value: N/A
Example:10

Integer/Expression

N/A

Specify the number of seconds for which the Snap must wait for the child pipeline to complete the runtime. If the child pipeline does not complete the runtime before the timeout, the execution process stops and is marked as failed.

Retry limit

Default Value: N/A
Example: 3

Integer/Expression

N/A

Specify the maximum number of retry attempts that the Snap must make in the case of a failure. If the child pipeline does not execute successfully, an error document is written to the error view. If the child pipeline is not in a completed state, then it will retry. The pipeline failure at the application level could have various causes, including network failures.

This feature is incompatible with reusable executions. 

Retry is not supported if the input view is a Binary data type.

Retry interval

Default Value: N/A
Example: 10

String/Expression

N/A

Specify the minimum number of seconds the Snap must wait between two retry requests. A retry happens only when the previous attempt results in an error. 

This feature is incompatible with reusable executions. 

Retry is not supported if the input view is a Binary data type.

Snap Execution

Default Value
Example: Validate & Execute

Dropdown list

N/A

Select one of the following three modes in which the Snap executes. The available options are:

  • Validate & Execute. Performs limited execution of the Snap and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only. Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled. Disables the Snap and all Snaps that are downstream from it.

Troubleshooting

Error

Reason

Resolution

Account validation failed.

The Pipeline ended before the batch could complete execution because of a connection error.

Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero so the connection is always open.

Access pipelines in the Pipeline Catalog in the Designer to create a Child pipeline

You can now browse the pipeline catalog for the target child pipeline and then select, drag, and drop it into Canvas. The Designer automatically adds the child pipeline using the Pipeline Execute Snap. You can preview a child pipeline by hovering over a Pipeline Execute Snap while the parent pipeline is open on the Designer.

Return child pipeline output to the parent pipeline

A common use case for the Pipeline Execute Snap is to run a child Pipeline whose output is immediately returned to the parent pipeline for further processing. You can achieve this return with the following Pipeline design for the child Pipeline.

In this example, the document in this child Pipeline is sent to the parent Pipeline through output1 of the child pipeline. Any unconnected output view is returned to the parent Pipeline. You can use any Snap that completes execution in this way.

Execution States

When a Pipeline Execute Snap activates its child pipeline, you can view the status as it executes on the SnapLogic Designer canvas.

A child pipeline also reports its execution to the parent pipeline. In the Monitor overview and Dashboard, you can hover over the status shown in the Status column for a pipeline with a Pipeline Execute Snap, and the following messages are displayed for the following scenarios:

  • If the parent Pipeline has the status Failed shown in the Status column, the following message is displayed: One of the child Pipelines Failed.

  • If the parent Pipeline has the status Completed with Errors shown in the Status column, the following message is displayed: One of the child Pipelines completed with Errors.

These execution state messages apply even when the child pipeline does not appear on the Dashboard because of child pipeline execution limits.

The Monitor Execution overview does not include the Completed with Warnings status as a searchable status.

Examples

Run a Child pipeline multiple times

The PE_Multiple_Executions project demonstrates how to configure the Pipeline Execute Snap to execute a child pipeline multiple times. The project contains the following pipelines:

  • PE_Multiple_Executions_Child: This is a simple child pipeline that writes out a document with a static string and the number of input documents received by Snap.

  • PE_Multiple_Executions_NoReuse_Parent: This parent pipeline executes the PE_Multiple_Executions_Child Pipeline five times. You can save the Pipeline to examine the output documents. Note that the output contains a copy of the original document, and the $inCount field is always set to one because the Pipeline was separately executed five times.

  • PE_Multiple_Executions_Reuse_Parent: This parent Pipeline executes the PE_Multiple_Executions_Child Pipeline once and feeds the child Pipeline execution five documents. You can save the Pipeline to examine the output documents. Note that the output does not contain a copy of the original document, and the $inCount field goes up for each document because the same Snap instance is being used to process each document.

  • PE_Multiple_Executions_UltraSplitAggregate_Parent: This parent Pipeline is an example of using Snaps that are not Ultra-compatible in an Ultra Pipeline. It can be turned into an Ultra Pipeline by removing the JSON Generator Snap at the head of the Pipeline and creating an Ultra Task.

  • PE_Multiple_Executions_UltraSplitAggregate_Child: A child Pipeline that splits an array field in the input document and sums the values of the $num field in the resulting documents.

Propagate a Schema Backward 

The project, PE_Backward_Schema_Propagation_Contacts, demonstrates the schema that suggests a feature of the Pipeline Execute Snap. It contains the following files:

  • PE_Backward_Schema_Propagation_Contacts_Parent

  • PE_Backward_Schema_Propagation_Contacts_Child

  • contact.schema (Schema file)

  • test.json (Output file)

The parent pipeline is shown below:

The child Pipeline is as shown below:

The Pipeline Execute Snap is configured as:

The following schema is provided in the JSON Formatter Snap. It has three properties - $firsName, $lastName, and $age. This schema is back propagated to the parent pipeline.

Pipeline-Execute-child-pipeline-schema.png

The parent Pipeline must be validated in order for the child Pipeline's schema to be back-propagated to the parent Pipeline. Below is the Mapper Snap in the parent pipeline: 

Notice that the Target Schema section shows the three properties of the schema in the child pipeline:

On execution, the data passed in the Mapper Snap will be written into the test.json file in the child Pipeline. The exported project is available in the Downloads section below.

Propagate Schema Backward and Forward

The project, PE_Backward_Forward_Schema_Propagation, demonstrates the Pipeline Execute Snap's capability of propagating schema in both directions – upstream and downstream. It contains the following Pipelines:

  • PE_Backward_Forward_Schema_Propagation_Parent

  • PE_Backward_Forward_Schema_Propagation_Child

The parent Pipeline is as shown below:

schema-parent.png

The Pipeline Execute Snap is configured to call the Pipeline schema-child. This child Pipeline consists of a Mapper Snap that is configured as shown below:

mapper-child.png

The Mapper Snaps upstream and downstream of the Pipeline Execute Snap: Mapper_InputSchemaPropagation and  Mapper_TargetSchemaPropagation are configured as shown below:

mapper-input-schema-propogation.png

 

mapper-target-schema-propogation.png

When the Pipeline is executed, data propagation takes place between the parent and child Pipeline:

  • The string expression $foo is propagated from the child Pipeline to the Pipeline Execute Snap.

  • The Pipeline Execute Snap propagates it to the upstream Mapper Snap (Mapper_InputSchemaPropagation), as visible in the Target Schema section. Here it is assigned the value 123.

  • This is passed from the Mapper to the Pipeline Execute Snap, which internally passes the value to the child Pipeline. Here, $foo is mapped to $bar. $baz is another string expression in the child Pipeline (assigned the value 2).

  • $bar, and $baz are propagated to the Pipeline Execute Snap and propagated forward to the downstream Mapper Snap (Mapper_TargetSchemaPropagation). This can be seen in the Input Schema section of the Mapper Snap.

Migrating from Legacy-Nested pipelines

The Pipeline Execute Snap can replace some uses of the Nested Pipeline mechanism and the ForEach and Task Execute Snaps. For now, this Snap only supports child Pipelines with unlinked document views (binary views are not supported). If these limitations are not a problem for your use case, read on to find out how to transition to this Snap and the advantages of doing so.

Nested Pipeline

Converting a Nested Pipeline will require the child pipeline to be adapted to have no more than a single unlinked document input view and no more than a single unlinked document output view. If the child pipeline can be made compatible, then you can use this Snap by dropping it on the canvas and selecting the child pipeline for the Pipeline property.

You will also want to enable the Reuse property to preserve the existing execution semantics of Nested pipelines. The advantages of using this Snap over Nested pipelines are:

  • Multiple executions can be started to process documents in parallel.

  • The Pipeline to execute can be determined dynamically by an expression.

  • The original document or binary data is attached to any output documents or binary data if reuse is not enabled.

ForEach

Converting the ForEach Snap to the Pipeline Execute Snap is a simple process. Select the pipeline to run and populate the parameters. The benefits of utilizing the Snap compared to the ForEach Snap are:

  • Documents or binary data fed into the Pipeline Execute Snap can be sent to the child Pipeline execution through an unlinked input view in the child.
    Documents or binary data sent from an unlinked output view in the child Pipeline execution is written out of the Pipeline Execute's output view.

  • The execution label can be changed.

  • The Pipeline to execute can be determined dynamically by an expression.

  • Executing a Pipeline does not require communication with the cloud servers. 

Task Execute

Converting a Task Execute Snap to a Pipeline Execute Snap is also straightforward because the properties are similar. 

To start, you only need to select the Pipeline you want to use; you no longer have to create a Triggered Task. If you set the Batch Size property in the Task Execute to one, then you will not want to enable the Reuse property. If the Batch Size was greater than one, then you should enable Reuse. The Pipeline parameters should be the same between the Snaps. The advantages of using this Snap to the Task Execute Snap are:

  • A task does not need to be created.

  • You can start multiple executions simultaneously to process documents in parallel.

  • You can process an unlimited number of documents that can be processed with a single execution (that is, no batch size).

  • The execution label can be changed.

  • Determine the pipeline to execute dynamically by an expression.

  • The original document is attached to any output documents if reuse is not enabled.

  • No requirement to communicate with the cloud servers to execute a pipeline.

Converting a pipeline that uses the Router Snap in "auto" mode can be done by moving the duplicated portions of the Pipeline into a new pipeline and then calling that Pipeline using a Pipeline Execute. After refactoring the Pipeline, you can adjust the "Pool Size" of the Pipeline Execute Snap to control how many operations are done in parallel. The advantages of using this Snap over an "Auto" Router are:

  • De-duplication of Snaps in the pipeline.

  • Adjustment of the level of parallelism is trivial simply change the Pool Size value.

Downloads

Important Steps to Successfully Reuse Pipelines

  1. Download the zip files, extract the files, and import the Pipelines into SnapLogic.

  2. Configure Snap accounts as applicable.

  3. Provide Pipeline parameters as applicable.

  File Modified
No files shared here yet.

Snap Pack History

 Click to view/expand
ReleaseSnap Pack VersionDateTypeUpdates
November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.
August 2024438patches28470 LatestAdded the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution workflow of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline, for scenarios where repeated processing is needed until a specific stop condition is met.
August 2024438patches28047 LatestEnhanced the Head and Tail Snaps (for the Number of documents and Document offset fields) to support pipeline parameters via expressions that enable you to reuse the pipelines.
August 2024main27765 Stable

When the Router Snap was the first Snap in a triggered task, the documents were not processed. Previously, the response contained a 200 status code but was empty. Now, it processes one document and provides non-empty output.

Behavior change:

Previously, the Router Snap generated no output without an input document. Now, the Snap generates an empty document when no input document is provided.

May 2024437patches26857 Latest

Enhanced the Pipeline Execute Snap to support expression values via pipeline parameters for the Batch size field that enables you to use intermediate pipelines for generic use cases.

May 2024

main26341

 

Stable

Updated and certified against the current SnapLogic Platform release.

March 2024436patches25576 LatestFixed an issue with the Pipeline Execute Snap where a condition resulted in the output documents not being read.
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.
August 2023434patches22576 Latest

Fixed an issue with the Pipeline Execute Snap that caused a null pointer exception during the retry when the input view was a Binary type. Retry is not supported for a Binary input view with this Snap.

August 2023

main22460

 


Stable

Updated and certified against the current SnapLogic Platform release.

May 2023

433patches22107

 Latest

Fixed an issue with the Pipeline Execute Snap that caused it to acknowledge the Ultra pipeline message before the message reached the last Snap in the pipeline. Because of the early acknowledgment, the last Snap in the pipeline failed and the response could not be sent back.

May 2023433patches21744 LatestFixed an issue with the Pipeline Execute Snap where a missing label produced a null pointer exception.
May 2023

433patches21196

 Latest

Fixed an issue with the Pipeline Execute Snap where no input view produced a null pointer exception.

May 2023433patches21040 LatestProvided a fix to ensure the Data Validator works with "unknown" data types.
May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023432patches20279 Latest

The Pipeline Execute Snap now returns errors from child pipelines with a unique identifier.

February 2023432patches19695 Latest
Fixed an issue with the Pipeline Execute Snap that was causing Unable to create a deep copy of the object errors.

As of  we have discovered an issue with this patch affecting the Pipeline Execute Snap. If you have pipelines leveraging the Pipeline Execute Snap, we recommend you NOT use this patch and instead use patch 432patches20279.

February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 StableUpgraded with the latest SnapLogic Platform release.
September 2022430patches18070 LatestThe Pipeline Execute Snap with binary output that is used in a low-latency feed Ultra Pipeline now works as expected. The requests are now acknowledged correctly.
August 2022main17386 StableUpgraded with the latest SnapLogic Platform release.

4.29

main15993

   

Stable

Upgraded with the latest SnapLogic Platform release.

4.28 Patch428patches15026 Latest

Fixed an issue with HTTP Router Snap where the Snap did not work as intended for a GET request that used a Cloud URL when the HTTP Router was the first Snap in the triggered task Pipeline.

4.28main14627 StableEnhanced the Filter Snap with the Null-safe access checkbox. Select this checkbox to consider the valid values in the filter expression and remove the missing or empty data instead of treating them as null.
4.27 Patch427patches13726
 
Latest

Enhanced the Filter Snap with the Null-safe access checkbox. Select this checkbox to consider the missing data or empty value as null when accessing the source path.

4.27main12833
 
StableEnhanced the Gate Snap with the Output Document Content field that enables you to indicate the count of input documents and the specific (first or last) document to include in the output.
4.26main11181
 
Stable

Added Binary Copy Snap to copy a binary stream to the Snap’s output views. You can use this Snap if you want to send the same information to multiple endpoints.

4.25 Patch425patches10571
 
Latest

Enhanced the performance of Binary Router Snap via more efficient IO utilization.

4.25main9554
 
Stable

Enhanced the HTTP Router Snap to support Ultra Pipelines.

4.24main8556
 
Stable

Added the Ignore empty stream checkbox to the Gate Snap Settings

4.23main7430
 
Stable

Enhances the Data Validator Snap by making the Constraint value field as optional under the Constraints.

4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21 Patch                     flow8855-Latest

Fixes the Binary Router Snap's failure in Ultra Pipelines by preventing a deep-copy (performing a copy of the original objects recursively) of binary data headers.

4.21snapsmrc542-StableUpgraded with the latest SnapLogic Platform release.
4.20snapsmrc535-Latest
  • Adds a new Snap, Gate, which enables you to consolidate multiple input streams into a single output document where each input stream is displayed as a label containing the list of records that were received by that input view.
  • The Pipeline Execute Snap now provides the following capabilities:
    • Specify the Snaplex on which to run child Pipelines. You can now select the Snaplex on which to run the child Pipelines through the Execute On and Snaplex Path fields. The default option is executing the child Pipeline on the same Snaplex node as the Parent, but you can also specify a different Snaplex through the drop-down list, which includes the option to manually enter a Snaplex name.
    • Configure retry logic for child Pipeline executions. You can now configure retry logic for child Pipeline executions though the following fields: Number of Retries, Retry Interval, and Timeout fields.
4.19snaprsmrc528-StableUpgraded with the latest SnapLogic Platform release.
4.18 Patchflow7743-Latest

Added a property Validate full JSON path to the Data Validator Snap to fix an issue wherein a field validation failure against a constraint does not produce an error.

4.18snapsmrc523-StableUpgraded with the latest SnapLogic Platform release.
4.17 Patchflow7637-Latest

Fixed an issue with the Data Validator Snap wherein the output fails to report all but one of the violations when there are multiple constraints of the Required type.

4.17 PatchALL7402-Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515-Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508-Latest

Pipeline Execute Snap: Added the capability to select either document (previously supported) or binary data (new) for your input and output Views.

4.15 Patchflow6263-Latest

Improved the error messages to include the source path of the constraint for data validation failures.

4.15snapsmrc500-StableUpgraded with the latest SnapLogic Platform release.
4.14snapsmrc490-StableUpgraded with the latest SnapLogic Platform release.
4.13snapsmrc486-StableUpgraded with the latest SnapLogic Platform release.
4.12 Patchflow4895-Latest

Fixed an issue in the Data Validator Snap that made multiple copies of lineage on the error path.

4.12snapsmrc480-Stable

Resolved an issue with the Union Snap that caused excessive CPU utilization.

4.11 Patchflow4461-Latest

Fixed an issue with the Pipeline Execute Snap that could cause the parent pipeline to freeze when the child pipeline fails.

4.11 Patchflow4292 -Latest

Resolved an issue with the Union Snap in Ultra Mode that spikes the CPU utilization to ~30% during startup of 120 instances.

4.11snapsmrc465-StableUpgraded with the latest SnapLogic Platform release.
4.10snapsmrc414-StableUpgraded with the latest SnapLogic Platform release.
4.9.0 Patchflow3094-Latest

Addressed an issue in Binary Router where the Snap did not wait until all threads were completed to write data to the output views.

4.9.0 Patchflow3320-Latest

Data Validator Snap - fixed Type constraint feature.

4.9snapsmrc405-StableUpgraded with the latest SnapLogic Platform release.
4.8.0snapsmrc398
Stable

Filter: Snap-aware error handling policy enabled for Spark mode. This ensures the error handling specified on the Snap is used.

4.7.0 Patchflow2598
Latest

Addressed a possible race condition when reusing pipelines with a Pipeline Execute Snap.

4.7.0flow2297
Stable

Fixed an issue with Ultra pipeline when invoking, using Pipeline Execute, a child pipeline that contains a non-Ultra compatible Snap.

4.7.0snapsmrc382
Stable
  • Recommendation: The Pipeline Execute Snap is meant to eventually replace ForEach and Task Execute. It is suggested you use Pipeline Execute for all new pipelines and update existing pipelines at your earliest convenience.
  • Updated the Pipeline Execute Snap with Execute during preview field.
4.6.0 Patchflow1909
Latest

Resolved an issue with Union Snap slowing down pipelines.

4.6.0snapsmrc362
Stable
  • The following Snaps now support error view in Spark mode: Copy, Union.
  • ForEach: The value for the Maximum instances property is limited to a maximum of 100. A value higher than 100 will automatically be limited down to 100 during pipeline execution.
  • Pipeline Execute: Snaplex property added to let you specify the Snaplex on which to run the child pipeline.
  • Resolved an issue in Task Execute Snap that caused an internal server error (status code: 500).
  • Resolved an issue in Pipeline Execute Snap that did not show parameter suggestions when the pipeline is selected from another project.
  • Resolved an issue in Copy Snap that showed errors on the UI while the Javascript was being executed.
  • Resolved an issue in Filter Snap that failed with "could not filter the document" if upstream data for ServiceNow Query.
4.5.1flow1574
Latest
  • Resolved an issue with Pipeline Execute not able to load pipeline selected from the pipeline picker property.
  • Enhanced error handling in Pipeline Execute for when Reuse is enabled and the pipeline path is an expression whose result is not constant.
  • Resolved an issue in Pipeline Execute not including the runtime ID and state in error documents.
4.5.0snapsmrc344
Stable
  • NEW! Pipeline Execute introduced in this release. This Snap provides a way to flexibly and efficiently execute pipelines and flowing documents into and out of these executions.

  • Resolved an issue in Task Execute Snap to ensure JAVA 7 and JAVA 8 Snaplex give the same output.

  • Resolved an issue with the Router Snap routing to the wrong output when used in Spark mode

4.4.0

Stable
  • Resolved an issue with Exit Snap hanging during validation if threshold is not met.

  • Spark support added to the Copy, Filter, Router, and Union Snaps.

4.3.2

Latest
  • Resolved an issue with Task Execute where java.math.BigInteger cannot be cast to java.lang.String exception if input document variable was a numeric type.

  • Resolved an issue with Task Execute failing with expression enabled.

  • Resolved an issue with data preview not working in the ForEach Snap.

  • Resolved an issue with Execute during Preview functionality broken in the ForEach Snap.

  • Resolved an issue with ForEach Snap not representing the output schema correctly during preview.

4.3.1

Latest

Resolved an issue with the ForEach poll interval being inconsistent.

4.3.0

Stable
  • In Data Validator, the pattern option now works with partial matches.

  • Exit Snap 

    • Error handling improved to handle messages that are non-strings.

    • Exit Snap now supports expressions in the Exit error message property.

  • ForEach Snap now generates output documents when run. Output documents will also be produced if a child pipeline called by the ForEach is manually stopped.

4.2.2

Latest
  • Resolved an issue in ForEach where a recursive pipeline in synchronous mode was unable to finish execution.

  • Resolved an issue in Data Validator not working for partial matches.



August 7, 2015 (2015.25/4.2.1)
  • NEW! Exit Snap introduced in this release. This Snap forces a pipeline to stop with a failed status if it receives more records than the user-defined threshold.

  • Resolved error in ForEach Snap where an exposed error view triggered a pipeline failure.



June 27, 2015 (2015.22)

The Router Snap now supports even distribution across all output views if no expressions are defined. With this change, Routes is no longer a required field.



June 6, 2015 (2015.20)
  • Task Execute should have Execute on preview
  • Directory Browser Snap did not render output data when running from "Windows" Groundplex.


May 15, 2015

ForEach: Error was not thrown on validation if the child pipeline was not provided.



May 2, 2015
  • Binary Router: enhanced error handling
  • Filter: bug fixes
  • Router: bug fixes


December 20, 2014

Updated Snaps may include new or changed functionality or be otherwise improved.

  • Data Validator: Type constraint added.
  • Union Snap: Preserve order was removed.


July/Summer 2014
  • NEW! Binary Router Snap introduced in this release.
  • NEW! Task Execute Snap introduced in this release.


June 30, 2014
  • Data Validator (Beta Release). This Snap validates incoming documents and their attributes against constraints you define.
  • ForEach: Addressed an issue with parallel execution not working properly.


April 2014

ForEach Snap updated to support Snaplex selection and pipeline parameters.



January 2014

NEW! ForEach Snap introduced in this release.



November 2013
  • NEW! Head Snap introduced in this release.
  • NEW! Tail Snap introduced in this release.


Initial Release (June 2013)
  • Router Snap introduced in this release.
  • Copy Snap introduced in this release.
  • Union Snap introduced in this release.
  • Filter Snap introduced in this release.

  • No labels