On this Page

Overview

Resumable Pipelines suspend the flow of data when an endpoint is down. In Resumable Mode, Pipelines are suspended instead of failed when an error is encountered, and Pipeline status is displayed on the SnapLogic Monitoring Dashboard. As such, Resumable Pipelines are fundamentally different in execution than standard Pipelines. In a Resumable Pipeline, every Snap runs to completion before any of its output documents are passed to the subsequent downstream Snap. Whereas in a standard Pipeline, a Snap outputs a document, which becomes the input to the subsequent downstream Snap even before the first Snap completes its document processing.

Executing a Resumable Pipeline saves the state of the Pipeline when hard exceptions occur and allows you to resume the Pipeline, running the original version of the Pipeline at the point of failure and not the latest version. Documents that have been processed successfully by a given Snap are not re-run.

With Resumable Pipelines, you can now achieve exactly-once-delivery. The Snaps in the Suspended Pipeline that have executed before suspension are green, while those Snaps that are yet to be executed turn orange. 

Prerequisite

Your Org must be subscribed to Resumable Pipelines to use this feature.

Workflow

Usage Guidelines and Limitations

• You must ensure sufficient disk space associated with the node hosting the Snaplex for suspensions to be stored. 
• You should not use consumer and subscriber Snaps, which are configured to run continuously, in Resumable Pipelines.
• Snaps used for bulk loads are not supported.
• You must disable batching in any Snaps that support batch jobs.
• Snaps that maintain their state during execution are supported. 
• Both document views and binary views are supported.
• The Resumed Pipeline is always the original version that was executed and does not reflect Pipeline changes since the original execution.
• The Pipe Execute Snap is not supported inside the Resumable Pipeline. However, a Resumable Pipeline can be a child Pipeline to a parent Pipeline with the Pipeline Execute Snap.

Enabling Resumable Mode

1. In SnapLogic Designer, select the target Pipeline and open the Properties menu. You can enable Resumable Mode in existing Pipelines.

2. On the Settings tab, select Standard for Type.
3. Click the Resumable Mode option.
   
4. Enter the necessary remaining information on the Settings tab, then click Save.

Monitoring Resumable Pipelines on the Dashboard

In SnapLogic Designer, if Snaps in your Resumable Pipeline turn orange, then your Pipeline is suspended. You can also configure a Task to send an email when it goes into the Suspended state. To view status of a Resumable Pipeline, go to the Dashboard.

The Status column displays the following additional states for Resumable Pipelines.

• Started. Indicates that Pipeline execution has been instantiated on the node.
• Completed. Indicates that the Pipeline has executed successfully.
• Failed. Indicates that Pipeline execution has encountered an error and is no longer executing.
• Suspending. Indicates that a Snap has encountered an error and the state of the execution is in the process of being saved.
• Suspended. Indicates that the resumable Pipeline is in suspension after a Snap failed.
• Resuming. Indicates that the state of the execution is being reloaded and instantiated on the node.
• Stopped. Occurs when the user manually stops Pipeline execution. 

You can resume suspended Pipelines after the endpoint is restored by selecting the Pipeline and clicking the Resume icon (). If you click the Stop icon (), then the execution state is deleted from the nodes and that execution is no longer be able to resume. Additionally, you cannot manually suspend a Resumable Pipeline.

Primary Use Case

The Resumable Pipeline feature is best implemented along the lines of the following use case. 

The following pipeline is an Ultra Pipeline with the Pipeline Execute Snap, which calls the Resumable Pipeline.

This parent Pipeline calls the following Resumable Pipeline, where two concurrent services are processing documents to be joined together and mapped. 

An error occurs in one of the Snap, so the remaining Snaps, which have not been executed, turn orange to indicate their suspended state in the Pipeline. 

The Dashboard shows one error document in the parent Pipeline indicating that an error occurred.

The Extra Details tab includes a history of states for the Resumable Pipeline.

FAQs

• Does a Suspended Pipeline restart from the beginning Snap and reprocess the data that has already been processed?
  No, when you resume a suspended Pipeline, the execution starts with the Snap that failed.
• Can Resumable Pipelines be applied to existing Pipelines that are currently running?

  Yes, as long as the pipeline fits in the constraints outlined in this document. You can open a Pipeline's properties and make it resumable.
• Are Resumable Pipelines applicable for all Snaplexes?

  Currently, only a standard-mode Snaplex supports Resumable Pipelines.
• How long does a Pipeline remain in a suspended state?
  Pipelines are initially in a suspended state for two weeks, then cleared from where they are stored on the Snaplex nodes. 
• Do Resumable Pipelines have an option for auto-resume after a set time limit and a set amount of retries?
  Currently, there is a Public Start API with which you can create your own auto-resume operations. 
• How is low volume defined?

  Currently, the volume, which can include both the quantity of documents and the capacity available, needs to fit onto the Snaplex file system where the document data is saved to a persisted state and allowed for at-least-once guaranteed delivery.
• Does the same version of document/binary data pass through a Resumable Pipeline after it's resumed?
  Yes, the original version of the Pipeline data runs on all subsequent attempts after the Pipeline is resumed.
• When a Resumable Pipeline is Suspended, does the Snap where the suspension occurs continue to stream data in an attempt to reach the endpoint?
  No, streaming is disabled so that all documents in a pipeline will complete one Snap at a time.
• Is the data from a Resumable Pipeline stored anywhere?
  Yes, the data remains in a persisted state in the Snaplex file system.
• Are there limitations when using Resumable Pipelines as Scheduled Tasks or Triggered Tasks?
  Yes, if the Do not start a new execution if one is already active checkbox is selected, a suspended execution is treated as "active". So, subsequent attempts to run the pipeline do not happen until the suspended execution is resumed or stopped. 
• Is there an effect on the processing of Pipelines when too many Resumable Pipelines have gone into a Suspended state?
  Yes, if the Snaplex file system is full, new executions do not occur for Resumable Pipelines.
• Do Resumable Pipelines work with Ultra Pipelines?
  Yes, Resumable Pipelines should be leveraged when it is at the end of an Ultra Pipeline workflow via the Pipeline Execute Snap. A Snap downstream of that Pipeline Execute Snap in the Ultra Pipeline should not fail.
• What happens when an Ultra Pipeline's child Resumable Pipeline goes into suspension?
  If the Resumable Pipeline goes into a suspended state, it returns an error to the Ultra Pipeline. The client calling the Ultra Pipeline receives that error and successfully acknowledges the message, thereby dequeuing from the FeedMaster. When the Suspended Pipeline is resumed, that Pipeline completes the execution. As a result, since that data flow has already completed, nothing returns to the parent Ultra Pipeline.

See Also

• Pipeline Monitoring API
• Ultra Task Pipelines