The SnapLogic control plane and data plane Control Plane and Data Plane together handle certain overload conditions on a Snaplex, which includes moving a Pipeline in a Queued state during Pipeline execution. When an a Pipeline execution is in the Queued state, the control plane initiates the Pipeline execution as resources become available on the Snaplex. You can also configure an Alert/Activity Notification for Snaplex instances to set resource thresholds that detect congestion.
The scheduling of Pipelines across nodes in a Snaplex is done based on the following criteria:
- Active threads
- CPU usage
- Memory usage
- Open file descriptors
based on an algorithm that is least-loaded, with priority on memory usage. If there are similarly loaded nodes, the algorithm randomizes the Pipeline execution across them.
You can utilize the Snaplex-level slot count field to control distributing the Pipelines to the nodes. The default value for slot count is 4000 slots, which correspond to active threads on a Snaplex node. For each Pipeline being prepared, SnapLogic generates an estimate of the slots required for executing the Pipeline, assuming that each Snap in the Pipeline and its child Pipelines consume a single thread. The Pipeline runtime info information dialog's Extra Details tab displays the estimate of the required slot count. To illustrate, if a Pipeline has 30 Snaps, and the Snaplex node has 3980 active threads, then the execution requests are Queued because sufficient slots are not available.
...
The actual threads a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps like such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, etc. can use a higher number of threads compared to other Snaps. The thread count varies based on factors like Snap configuration and data volume. Not all Snaps in a Pipeline process concurrently; the data processing depends on on how data streams through a given Pipeline. Further, the resources a Pipeline consumes can vary over time.
| Note |
|---|
|
...