On this Page

Overview

Resumable Pipelines suspend the flow of data when an endpoint becomes inaccessible. If an exception disables a target endpoint, the Resumable Pipeline's execution state is saved in the Snaplex nodes. After restoring connectivity to the target endpoint, you can resume the suspended Pipeline starting at the point of failure, so that successfully processed documents are not processed again.

Every Snap in a Resumable Pipeline runs to completion before any of its output documents are passed to the next downstream Snap. In contrast, a Snap in a standard Pipeline passes the document to the next downstream Snap, possibly even before the first Snap completes its document processing.

The SnapLogic Monitoring Dashboard displays the status of Pipelines. If a Pipeline is suspended, the Snaps that have completed execution are displayed in green, and the remaining Snaps are displayed in orange.

You can resume a suspended Pipeline from the SnapLogic Dashboard.

Prerequisite

Your Org must be subscribed to Resumable Pipelines to use this feature.

Workflow

Usage Guidelines

• Snaps that maintain their state during execution are supported. 
• Both document views and binary views are supported.
• You must ensure sufficient disk space associated with the node hosting the Snaplex for suspensions to be stored. 
• You should not use consumer and subscriber Snaps, which are configured to run continuously, in Resumable Pipelines.
• In the child-parent pipeline design, a Resumable Pipeline can not be the parent Pipeline. However, a Pipeline Execute Snap in a parent Pipeline can call a child Resumable Pipeline.

Limitations

• Snaps used for bulk loads are not supported.
• Batching is not supported. You must disable batching in any Snaps that support batch jobs.
• The Resumed Pipeline is always the original version that was executed and does not reflect Pipeline changes since the original execution.

Enabling Resumable Mode

1. In SnapLogic Designer, select the target Pipeline and open the Properties menu. You can enable Resumable Mode in existing Pipelines.

2. On the Settings tab, select Standard for Type.
3. Click the Resumable Mode option.
   
4. Enter the necessary remaining information on the Settings tab, then click Save.

Monitoring Resumable Pipelines on the Dashboard

In SnapLogic Designer, if Snaps in your Resumable Pipeline turn orange, then your Pipeline is suspended. You can also configure a Task to send an email when it goes into the Suspended state. To view status of a Resumable Pipeline, go to the Dashboard.

The Status column displays the following additional states for Resumable Pipelines.

• Started. Indicates that Pipeline execution has been instantiated on the node.
• Completed. Indicates that the Pipeline has executed successfully.
• Failed. Indicates that Pipeline execution has encountered an error and is no longer executing.
• Suspending. Indicates that a Snap has encountered an error and the state of the execution is in the process of being saved.
• Suspended. Indicates that the resumable Pipeline is in suspension after a Snap failed.
• Resuming. Indicates that the state of the execution is being reloaded and instantiated on the node.
• Stopped. Occurs when the user manually stops Pipeline execution. 

You can resume suspended Pipelines after the endpoint is restored by selecting the Pipeline and clicking the Resume icon (). If you click the Stop icon (), then the execution state is deleted from the nodes and that execution is no longer be able to resume. Additionally, you cannot manually suspend a Resumable Pipeline.

Use a Pipeline Execute Snap inside an Ultra Pipeline to call a Resumable Pipeline

The Resumable Pipeline feature is best implemented along the lines of the following use case. 

A Pipeline Execute Snap, inside an Ultra Pipeline Task, calls the Resumable Pipeline.

This Resumable Pipeline contains two concurrent services are processing documents to be joined together and mapped. 

An error occurs in one of the Snaps, so the remaining Snaps, which have not been executed, turn orange to indicate their suspended state in the Pipeline. 

The Dashboard shows one error document in the Pipeline Execute Snap inside the parent Pipeline that indicates an error occurred.

The Extra Details tab includes a history of states for the Resumable Pipeline.

FAQs

• Does a Suspended Pipeline restart from the beginning Snap and reprocess the data that has already been processed?
  No, when you resume a suspended Pipeline, the execution starts with the Snap that failed.
• Can Resumable Pipelines be applied to existing Pipelines that are currently running?

  Yes, as long as the pipeline fits the usage guidelines in this document. You can open a Pipeline's properties and make it resumable.
• How long does a Pipeline remain in a suspended state?
  Pipelines are initially in a suspended state for two weeks, then cleared from where they are stored on the Snaplex nodes. 
• What happens when an Ultra Pipeline's child Resumable Pipeline goes into suspension?
  When the child Resumable Pipeline is suspended, the platform returns the status to the Pipeline Execute Snap in the parent Pipeline. You can configure the Pipeline Execute Snap to have an open error view to handle the child Resumable Pipeline status: continue or fail. When the suspended child Pipeline is resumed, that Pipeline completes the execution. As a result, since that data flow has already been completed, nothing returns to the parent Ultra Pipeline.
• Do Resumable Pipelines have an option for auto-resume after a set time limit and a set amount of retries?
  Currently, there is a Public Start API with which you can create your own auto-resume operations. 
• Does the same version of document/binary data pass through a Resumable Pipeline after it's resumed?
  Yes, the original version of the Pipeline data runs on all subsequent attempts after the Pipeline is resumed.
• When a Resumable Pipeline is Suspended, does the Snap where the suspension occurs continue to stream data in an attempt to reach the endpoint?
  No, streaming is disabled so that all documents in a pipeline will complete one Snap at a time.
• Is the data from a Resumable Pipeline stored anywhere?
  Yes, the data remains in a persisted state in the Snaplex file system.
• Are there limitations when using Resumable Pipelines as Scheduled Tasks or Triggered Tasks?
  Yes, if the Do not start a new execution if one is already active checkbox is selected, a suspended execution is treated as "active". So, subsequent attempts to run the pipeline do not happen until the suspended execution is resumed or stopped. 
• Is there an effect on the processing of Pipelines when too many Resumable Pipelines have gone into a Suspended state?
  Yes, if the Snaplex file system is full, new executions do not occur for Resumable Pipelines.
• Do Resumable Pipelines work with Ultra Pipelines?
  Yes, Resumable Pipelines should be leveraged when it is at the end of an Ultra Pipeline workflow via the Pipeline Execute Snap. A Snap downstream of that Pipeline Execute Snap in the Ultra Pipeline should not fail.

See Also

• Pipeline Monitoring API
• Ultra Tasks