Pipeline Queueing

The SnapLogic Control Plane and Data Plane together handle certain overload conditions on a Snaplex, which includes moving a Pipeline in a Queued state during Pipeline execution. When a Pipeline execution is in the Queued state, the control plane initiates the Pipeline execution as resources become available on the Snaplex. You can also configure an Alert/Activity Notification for Snaplex instances to set resource thresholds that detect congestion.

The scheduling of Pipelines across nodes in a Snaplex is based on an algorithm that is least-loaded, with priority on memory usage. If there are similarly loaded nodes, the algorithm randomizes the Pipeline execution across them.

You can utilize the Snaplex-level slot count field to control distributing the Pipelines to the nodes. The default value for slot count is 4000 slots, which correspond to active threads on a Snaplex node. For each Pipeline being prepared, SnapLogic generates an estimate of the slots required for executing the Pipeline, assuming that each Snap in the Pipeline and its child Pipelines consume a single thread. The Pipeline runtime information dialog's Extra Details tab displays the estimate of the required slot count. To illustrate, if a Pipeline has 30 Snaps, and the Snaplex node has 3980 active threads, then the execution requests are Queued because sufficient slots are not available.

The actual threads a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, can use a higher number of threads compared to other Snaps. The thread count varies based on factors like Snap configuration and data volume. Not all Snaps in a Pipeline process concurrently; the data processing depends on how data streams through a given Pipeline. Further, the resources a Pipeline consumes can vary over time.

A Pipeline does not remain in the Queued state indefinitely and times out after a specified time. A scheduled Pipeline times out before the next scheduled execution or four hours, whichever comes first.
Not all methods of Pipeline execution result in a Pipeline being in the Queued state. The Queued state applies only to Pipeline executions started by a Scheduled Task.

Pipeline executions can fail immediately if the Snaplex is overloaded; this condition triggers the alert notification if configured. Triggered Task executions retry for a minute before returning a failure to the requesting client.

You can configure slots per Snaplex. For more information, see Creating a Snaplex.