Versions Compared

Old Version 12

changes.mady.by.user John Brinckwirth

Saved on

compared with

New Version Current

changes.mady.by.user Lakshmi Manda

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
absoluteUrltrue

Overview

Ultra Pipeline Tasks require consideration and preparation of your network depending on the purpose of the Ultra Pipeline.   The following deployment architectures cover the most common scenarios to for setting up your environment to run Ultra TasksPipelines in Ultra mode.

One component present in most configurations is the FeedMaster, which brokers the queues between the execution nodes and Clients. The FeedMaster receives HTTP requests and turns them into documents and sends them to nodes for processing. The response from the node sends it to the FeedMaster, which in turn returns it as a response to incoming requests.

No FeedMaster

Headless Ultra Pipelines do not require a FeedMaster, so setting up this Snaplex is simpler than other configurations.

  • Use this configuration if you already have a data source that

    providers

    provides a listener-type of

    interface

    interfaces, such as JMS or file poller.

    Info

    The following are the Listener Snaps that can be used in the headless Ultra pipelines:


  • The Ultra Pipeline functionality ensures that N number of listener Pipelines are always running. Since the source (or listener-type interface) is already a listener, you do not need to add a FeedMaster.
  • This type of Ultra Pipeline does not provide a REST API interface for the Pipeline, ; this is just providing Always-On Pipeline functionality.
  • One of the features of Ultra is the capability to have instances of Pipelines which that are Always On, it does not use the FeedMaster infrastructure at all.
  • When you use a JMS listener in a standard Pipeline execution, an individual instance of the Pipeline just runs, waiting for input documents to arrive from the JMS queue. Each instance is individual.
  • When you define the same as an Ultra type of Ultra Pipeline as a Task, you can ask for N instances to be instantiated across the Snaplex, which SnapLogic monitors and restarts automatically, ensuring that the full number of instances are continuously running.

One FeedMaster

Configured 

This configuration allows customers to build a low latency REST interface for talking to communicating with any kind type of endpoint. It provides low latency and reliability as compared to a Triggered Task, which has dependency is dependent on the control plane). A FeedMaster can be  

  • A FeedMaster is installed in front of the
execution node
  • JCC node (also known as the execution node) for any return-and-request Ultra
Pipeline Tasks
  • Pipelines.
  • Plain HTTP/S requests can be fed into a pipeline through a FeedMaster that is installed as part of a Snaplex. The HTTP request is turned into a document. 
  • This document is sent to the Pipeline's
unlinked
  • unconnected input, which is turned into the HTTP response to the original request
.

Standard Network Configuration - 2 FeedMasters and 1 Load Balancer

This configuration can be deployed for Groundplexes.  

The following diagram provides an overview of the standard network architecture required for using Ultra Pipeline Tasks, including the communication between the various components with a focus on the network ports
  • .
Image Removed

  • The FeedMaster uses the following ports:

    • 8084: The FeedMaster's HTTPS port. 
    • 8089 - The FeedMaster's embedded ActiveMQ broker TLS (SSL) port.
    Note
    titleFeedMaster Ports

    The machine hosting the FeedMaster must have those ports available on the local firewall. 

High Availability Configuration - 4



Standard Configuration—Two FeedMasters and

2

One Load

Balancers

Balancer

The following diagram provides an overview of the High Availability standard network architecture required for Ultra Pipeline Tasks, including the communication between the various components with a focus on the network ports.
Image Added



Disaster Recovery Configuration—Four FeedMasters and Two Load Balancers

The following diagram provides an overview of the Disaster Recovery architecture for Ultra Pipelines. 

Image Modified

Configuration on DNS Server

  • If you are deploying the Snaplex in an Active-Active configuration, set Route53  Route53 to support Active-Active pairs across the F5 LoadBalancers (F5-1E and F5-1W)load balancers (East and West).
  • If you are deploying the Snaplex in an Active-Standby configuration, set Route53 s across the F5 LoadBalancers (F5-1E and F5-1W)load balancers (East and West).

Deployment of Tasks

  • All tasks deployed on Ultra East Tasks deployed in the Eastern region should have the same unique pre/postfixAll tasks deployed on Ultra West ; likewise, all Tasks deployed in the Western region should have the same unique pre/postfix.
  • For example we have a Task "<task , the Task, <task_name>_TaskOrgNameEast" that , is deployed to "Ultra Plex East" -Where "TaskOrgNameEast" UltraPlex East, where TaskOrgNameEast is a unique string that is present as the pre/postfix for all Tasks deployed on Ultra Plex UltraPlex East. For example we have a Task "
  • Similarly, the Task, <task name>_TaskOrgNameWest" ,that is deployed to "Ultra Plex West" - Where "TaskOrgNameWest" UltraPlex West, where TaskOrgNameWest is a unique string that is present as the  pre/postfix for all Tasks deployed on Ultra Plex UltraPlex West.
  • Both Tasks must have the same Alias.

Configuration Guidelines for F5 a Load Balancer

  • Use an F5 LoadBalancer or API gateway that supports URI rewrites.
  • Do not use the AWS elastic load balancer because it does not support URI rewrite. 
  • "F5-1E" is configured to replace "_TaskOrgNameWest" in the URI to "_TaskOrgNameEast""F5-1W" is configured to replace "_TaskOrgNameEast" in the URI to "_TaskOrgNameWest"Load balancers require FeedMaster port information.
  • Configure an Alias on both Tasks. The Alias Name is used for the Task URL.
  • Use the HealthZ URL (https://<HOSTNAME>:8084/healthz) to monitor communication between the load balancers and the FeedMasters. For details, see Deploying a FeedMaster Node.


See Also