The SnapLogic control plane and data plane together handle certain overload conditions on a Snaplex, which includes moving a Pipeline in a Queued state during Pipeline execution. When an execution is in the Queued state, the control plane initiates the Pipeline execution as resources become available on the Snaplex. You can also configure an Alert/Activity Notification for Snaplex instances to set resource thresholds that detect congestion.
The scheduling of Pipelines across nodes in a Snaplex is done based on the following criteria:
- Active threads
- CPU usage
- Memory usage
- Open file descriptors
You can utilize the Snaplex-level slot count field to control distributing the Pipelines to the nodes. The default value for slot count is 4000 slots, which correspond to active threads on a Snaplex node. For each Pipeline being prepared, SnapLogic generates an estimate of the slots required for executing the Pipeline assuming that each Snap in the Pipeline and its child Pipelines consume a single thread. The Pipeline runtime info dialog's Extra Details tab displays the estimate of the required slot count. To illustrate, if a Pipeline has 30 Snaps, and the Snaplex node has 3980 active threads, then the execution requests are Queued because sufficient slots are not available.
The actual threads a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps like Pipeline Execute, Bulk loaders, Snaps performing input/output, etc. can use a higher number of threads compared to other Snaps. The thread count varies based on factors like Snap configuration and data volume. Not all Snaps in a Pipeline process concurrently; the data processing depends on data streams through a given Pipeline. Further, the resources a Pipeline consumes can vary over time.
Note |
---|
|
...