The SnapLogic control plane and data plane together handle certain overload conditions on a Snaplex, which includes moving a Pipeline in a Queued state during Pipeline execution. When an execution is in the Queued state, the control plane initiates the Pipeline execution as resources become available on the Snaplex. You can also configure an Alert/Activity Notification for Snaplex instances to set resource thresholds that detect congestion.
The scheduling of Pipelines across nodes in a Snaplex is done based on the following criteria:
- Active threads
- CPU usage
- Memory usage
- Open file descriptors
You can utilize the Snaplex-level slot count field to control distributing the Pipelines to the nodes. The default value for slot count is 4000 slots, which correspond to active threads on a Snaplex node. For each Pipeline being prepared, SnapLogic generates an estimate of the slots required for executing the Pipeline assuming that each Snap in the Pipeline and its child Pipelines consume a single thread. The Pipeline runtime info dialog's Extra Details tab displays the estimate of the required slot count. To illustrate, if a Pipeline has 30 Snaps, and the Snaplex node has 3980 active threads, then the execution requests are Queued because sufficient slots are not available.
The actual threads a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps like Pipeline Execute, Bulk loaders, Snaps performing input/output, etc. can use a higher number of threads compared to other Snaps. The thread count varies based on factors like Snap configuration and data volume. Not all Snaps in a Pipeline process concurrently; the data processing depends on data streams through a given Pipeline. Further, the resources a Pipeline consumes can vary over time.
- A Pipeline does not remain in the Queued state indefinitely and times-out after a specified time. A scheduled Pipeline times-out before the next scheduled execution or four hours, whichever comes first.
- Not all methods of execution result in a Pipeline being in the Queued state. The Queued state applies only to executions started by a Scheduled Task.
Pipeline executions can fail immediately if the Snaplex is overloaded; this condition triggers the alert notification if configured. Triggered Task executions retry for a minute before returning a failure to the requesting client.
You can configure slots per Snaplex. For more information, see Creating a Snaplex.