In this Article
Performance of an Ultra Pipeline largely depends on the response times of the end-system applications to which the Task connects. An Ultra Pipeline containing a large number of high latency endpoint Snaps can result in a congestion of documents, building all the way up through the upstream Snaps to the FeedMaster, until the FeedMaster queue can no longer hold the messages. You can avoid this situation by either creating multiple instances of the Ultra Pipeline or by using the Pipeline Execute with the Reuse functionality enabled and the Pool Size field set to greater than one. Multiple instances of an Ultra Pipeline ensures that even if one instance is slow, others are available to consume documents and keep the FeedMaster queue flowing; a situation you can manage with the Max In-Flight setting on the Task form.
Likewise, you can use a Router Snap in each instance of the pipeline to distribute the documents across multiple endpoint Snaps and improve the performance and add parallel processing capability to an instance. This functionality is in addition to the built-in parallel computation capability of a pipeline which implies that at a given point in time, each Snap in a pipeline is processing a different document.
Scaling can be attained by increasing the number of instances in an Ultra Pipeline. The total number of instances required for an Ultra Pipeline is a direct function of the expected response time: the resource utilization of the node (when a single instance of the Task is running) and the functional load on the Snaplex from other pipeline runs. When execution nodes are highly utilized, adding more execution nodes allows the instances of the Task to be distributed horizontally and scaled out across the Snaplex.
High availability Architecture
To avoid service disruption and enable high availability, use a load balancer with two FeedMasters and two execution nodes as the minimum architecture for Ultra Pipeline setup. This architecture also avoids a single point of failure from the FeedMaster or execution node.
Monitoring Tasks in SnapLogic Manager
Each Task is listed in the Manager > Project > Task. To view the number of documents received or processed by each instance of the Task, a drop down menu lists the details of the Task.
Use the Details menu to view the Pipeline Execution stats and monitor documents received by each Snap in the Ultra Pipeline instance. Because the detailed view lists only the documents received by that instance of the Task, a more advanced monitoring plan might be required in case of multiple instances to get an overview of the documents, queues, and subscribers.