The SnapLogic control plane and data plane were improved in the Winter 2016/4.4 release to more gracefully Control Plane and Data Plane together handle certain overload conditions on a Snaplex. This change introduces a new "Queued" state for pipeline executions and adds properties to the Snaplex configuration to set the resource thresholds used to detect an overload.
When an execution is in the Queued state, the control plane will try to start the execution when resources become available on the Snaplex.
| Note |
|---|
|
Pipeline executions started by the Designer will fail immediately if the Snaplex is overloaded so that you are alerted of the overload. Triggered Task executions will , which includes moving a Pipeline in a Queued state during Pipeline execution. When a Pipeline execution is in the Queued state, the control plane initiates the Pipeline execution as resources become available on the Snaplex. You can also configure an Alert/Activity Notification for Snaplex instances to set resource thresholds that detect congestion.
The scheduling of Pipelines across nodes in a Snaplex is based on an algorithm that is least-loaded, with priority on memory usage. If there are similarly loaded nodes, the algorithm randomizes the Pipeline execution across them.
You can utilize the Snaplex-level slot count field to control distributing the Pipelines to the nodes. The default value for slot count is 4000 slots, which correspond to active threads on a Snaplex node. For each Pipeline being prepared, SnapLogic generates an estimate of the slots required for executing the Pipeline, assuming that each Snap in the Pipeline and its child Pipelines consume a single thread. The Pipeline runtime information dialog's Extra Details tab displays the estimate of the required slot count. To illustrate, if a Pipeline has 30 Snaps, and the Snaplex node has 3980 active threads, then the execution requests are Queued because sufficient slots are not available.
The actual threads a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, can use a higher number of threads compared to other Snaps. The thread count varies based on factors like Snap configuration and data volume. Not all Snaps in a Pipeline process concurrently; the data processing depends on how data streams through a given Pipeline. Further, the resources a Pipeline consumes can vary over time.
| Note |
|---|
|
Pipeline executions can fail immediately if the Snaplex is overloaded; this condition triggers the alert notification if configured. Triggered Task executions retry for a minute before returning a failure to the requesting client.
Slot allocation is configured You can configure slots per Snaplex. See the Update Snaplex dialog for these optionsFor more information, see Creating a Snaplex.