Resumable Pipelines

On this Page

Overview

Resumable Pipelines suspend the flow of data when an endpoint becomes inaccessible. If an exception disables a target endpoint, the Resumable Pipeline's execution state is saved in the Snaplex nodes. After restoring connectivity to the target endpoint, you can resume the suspended Pipeline starting at the point of failure, so that successfully processed documents are not processed again.

Every Snap in a Resumable Pipeline runs to completion before any of its output documents are passed to the next downstream Snap. In contrast, a Snap in a standard Pipeline passes the document to the next downstream Snap, possibly even before the first Snap completes its document processing.

The SnapLogic Monitoring Dashboard displays the status of Pipelines. If a Pipeline is suspended, the Snaps that have completed execution are displayed in green, and the remaining Snaps are displayed in orange.

You can resume a suspended Pipeline from the SnapLogic Dashboard.



Prerequisite

Your Org must be subscribed to Resumable Pipelines to use this feature.


Workflow


Usage Guidelines and Limitations

  • You must ensure sufficient disk space associated with the node hosting the Snaplex for suspensions to be stored. 
  • You should not use consumer and subscriber Snaps, which are configured to run continuously, in Resumable Pipelines.
  • Snaps used for bulk loads are not supported.
  • You must disable batching in any Snaps that support batch jobs.
  • Snaps that maintain their state during execution are supported. 
  • Both document views and binary views are supported.
  • The Resumed Pipeline is always the original version that was executed and does not reflect Pipeline changes since the original execution.
  • The Pipe Execute Snap is not supported inside the Resumable Pipeline. However, a Resumable Pipeline can be a child Pipeline to a parent Pipeline with the Pipeline Execute Snap.


Enabling Resumable Mode

  1. In SnapLogic Designer, select the target Pipeline and open the Properties menu. You can enable Resumable Mode in existing Pipelines.

  2. On the Settings tab, select Standard for Type.
  3. Click the Resumable Mode option.
  4. Enter the necessary remaining information on the Settings tab, then click Save.


By default, the Error Pipeline option is disabled and incompatible with Resumable Pipelines.

Monitoring Resumable Pipelines on the Dashboard

In SnapLogic Designer, if Snaps in your Resumable Pipeline turn orange, then your Pipeline is suspended. You can also configure a Task to send an email when it goes into the Suspended state. To view status of a Resumable Pipeline, go to the Dashboard.

The Status column displays the following additional states for Resumable Pipelines.

  • Started. Indicates that Pipeline execution has been instantiated on the node.
  • Completed. Indicates that the Pipeline has executed successfully.
  • Failed. Indicates that Pipeline execution has encountered an error and is no longer executing.
  • Suspending. Indicates that a Snap has encountered an error and the state of the execution is in the process of being saved.
  • Suspended. Indicates that the resumable Pipeline is in suspension after a Snap failed.
  • Resuming. Indicates that the state of the execution is being reloaded and instantiated on the node.
  • Stopped. Occurs when the user manually stops Pipeline execution. 

You can resume suspended Pipelines after the endpoint is restored by selecting the Pipeline and clicking the Resume icon (). If you click the Stop icon (), then the execution state is deleted from the nodes and that execution is no longer be able to resume. Additionally, you cannot manually suspend a Resumable Pipeline.


Primary Use Case

The Resumable Pipeline feature is best implemented along the lines of the following use case. 

The following pipeline is an Ultra Pipeline with the Pipeline Execute Snap, which calls the Resumable Pipeline.


This parent Pipeline calls the following Resumable Pipeline, where two concurrent services are processing documents to be joined together and mapped. 

An error occurs in one of the Snap, so the remaining Snaps, which have not been executed, turn orange to indicate their suspended state in the Pipeline. 

The Dashboard shows one error document in the parent Pipeline indicating that an error occurred.

The Extra Details tab includes a history of states for the Resumable Pipeline.


FAQs

  • Does a Suspended Pipeline restart from the beginning Snap and reprocess the data that has already been processed?
    No, when you resume a suspended Pipeline, the execution starts with the Snap that failed.
  • Can Resumable Pipelines be applied to existing Pipelines that are currently running?

    Yes, as long as the pipeline fits in the constraints outlined in this document. You can open a Pipeline's properties and make it resumable.
  • Are Resumable Pipelines applicable for all Snaplexes?

    Currently, only a standard-mode Snaplex supports Resumable Pipelines.
  • How long does a Pipeline remain in a suspended state?
    Pipelines are initially in a suspended state for two weeks, then cleared from where they are stored on the Snaplex nodes. 
  • Do Resumable Pipelines have an option for auto-resume after a set time limit and a set amount of retries?
    Currently, there is a Public Start API with which you can create your own auto-resume operations. 
  • How is low volume defined?
    Currently, the volume, which can include both the quantity of documents and the capacity available, needs to fit onto the Snaplex file system where the document data is saved to a persisted state and allowed for at-least-once guaranteed delivery.
  • Does the same version of document/binary data pass through a Resumable Pipeline after it's resumed?
    Yes, the original version of the Pipeline data runs on all subsequent attempts after the Pipeline is resumed.
  • When a Resumable Pipeline is Suspended, does the Snap where the suspension occurs continue to stream data in an attempt to reach the endpoint?
    No, streaming is disabled so that all documents in a pipeline will complete one Snap at a time.
  • Is the data from a Resumable Pipeline stored anywhere?
    Yes, the data remains in a persisted state in the Snaplex file system.
  • Are there limitations when using Resumable Pipelines as Scheduled Tasks or Triggered Tasks?
    Yes, if the Do not start a new execution if one is already active checkbox is selected, a suspended execution is treated as "active". So, subsequent attempts to run the pipeline do not happen until the suspended execution is resumed or stopped. 
  • Is there an effect on the processing of Pipelines when too many Resumable Pipelines have gone into a Suspended state?
    Yes, if the Snaplex file system is full, new executions do not occur for Resumable Pipelines.
  • Do Resumable Pipelines work with Ultra Pipelines?
    Yes, Resumable Pipelines should be leveraged when it is at the end of an Ultra Pipeline workflow via the Pipeline Execute Snap. A Snap downstream of that Pipeline Execute Snap in the Ultra Pipeline should not fail.
  • What happens when an Ultra Pipeline's child Resumable Pipeline goes into suspension?
    If the Resumable Pipeline goes into a suspended state, it returns an error to the Ultra Pipeline. The client calling the Ultra Pipeline receives that error and successfully acknowledges the message, thereby dequeuing from the FeedMaster. When the Suspended Pipeline is resumed, that Pipeline completes the execution. As a result, since that data flow has already completed, nothing returns to the parent Ultra Pipeline.