Deploying a FeedMaster Node

Owned by John Brinckwirth
Last updated: Oct 23, 2023Version comment

Overview

A FeedMaster is a specialized type of Snaplex node that controls the distribution of Ultra task requests across the JCC nodes inside the Snaplex. It is a Java Virtual Machine (JVM) that acts as an interface between the JCC nodes, which execute the Pipelines, and the client. Because FeedMaster nodes in a Cloudplex are deployed by SnapLogic, this article addresses FeedMasters in a Groundplex (self-managed Snaplex), which your organization sets up.


Prerequisites

  • To use a FeedMaster node for Ultra Task Pipelines, your Org must be subscribed to the Ultra feature.
  • The FeedMaster requires a Linux or Windows server.
  • The FeedMaster should be installed physically in a location near the Snaplex instances and the data passed by the Pipeline to support low-latency processing.
  • Default ports for components:
    • JCC HTTP/S—8081
    • FeedMaster HTTPS—8084
    • FeedMaster Broker—8089
    • FeedMaster HTTP—8090
  • Snaplex HTTP Interface
    • Local host—127.0.0.1
    • Any network—0.0.0.0

If other resources are using the specified ports, you can change the port number to the component on the Node Properties tab.


Deploying a FeedMaster

For details about server requirements, see Snaplex Installation on Linux for Groundplex installation and Groundplex Deployment Planning for specific configurations.

Health Z URL

You can configure more than one FeedMaster. In that configuration, you put a load balancer in front of the FeedMasters so that the requests are automatically distributed. The /healthz URL can be used by the load balancer to check for the health status of the FeedMaster.

To deploy a FeedMaster:

  1. In SnapLogic Manager, navigate to the target Project, then click  and select Snaplex from the drop-down menu.

  2. Enter the information in the Create a Snaplex dialog on the Settings tab.

  3. Click the Node Properties tab; under Snaplex node types, click , then enter the name of the hosting server in the Hostname field.


  4. From the Server type drop-down list, select FeedMaster. 
    Alternatively, you can create the Snaplex node first, then specify the Snaplex node-type in the SnapLogic global.properties file by setting server_type=feed_master.

    Specifying a DNS Host

    If specifying a DNS host server name or other custom host name, then enter the hostname in the Snaplex properties to define FeedMaster node. The actual host name is the same one returned by the hostname command on your Linux or Windows machine.

  5. Click Update.

In the SnapLogic Dashboard, the FeedMaster Node is designated by the  icon.


Storage Limits

Because the FeedMaster node has to queue up requests, a certain amount of storage is required. The requests stay in the queue and take up space on disk until the message has been processed by the Pipeline.

  • The default space for queued messages is 10GB.

  • The maximum storage which can be configured depends on the disk space available.

  • As the storage limit is reached, the FeedMaster rejects new requests by failing with HTTP 420 error message, Rate limit exceeded.

  • If one of the upstream services used by the Ultra Pipeline is slow, that can cause the queue to grow. Setting the storage limit higher allows for more requests to be queued up, but the higher setting also delays the clients from being notified about the upstream slowness.

  • The storage space used is for each FeedMaster node. If your deployment consists of multiple FeedMasters, each request takes up space on one of the FeedMaster nodes.

  • All Ultra Pipelines configured on a Snaplex share this disk space. The jcc.broker.disk_limit property specifies the maximum space used to store queued messages on disk when the message is being processed. The default value is 10GB.

    • If your Snaplex uses the slpropz configuration method, you can configure this value in Snaplex > Node Properties tab in Manager by specifying the limit in bytes in the Global properties field set:
      jcc.broker.disk_limit=20971520

    • If your Snaplex does not use the slpropz configuration method, you must add the jcc.broker.disk_limit value as an entry in the global.properties configuration file.

  • You can also change the storage location for the broker files using the following properties:
    jcc.broker_data_dir=/mnt/data
    jcc.broker_sched_dir=/mnt/data
    jcc.broker_tmp_dir=/mnt/data

  • The default location is $SL_ROOT/run/broker. Ensure that the mount you use for the broker files has sufficient disk space as per the disk_limit configuration. The snapuser user should have full access to the data location.

When running multiple Tasks on one Snaplex/FeedMaster, consider the following:

  • If a Task is running slow, the FeedMaster starts queueing the requests, and clients sending the requests begin receiving errors. Other Tasks on the Snaplex should not be affected.
  • If more than half of the Tasks running on the Snaplex are slow, then the requests to the slow Tasks begin receiving errors, but the other Tasks should not be affected.

In some cases, in the latter scenario, all Tasks may be affected, even though some isolation between Task performance is maintained by the FeedMaster. To ensure complete isolation, we recommend that you use separate Snaplex instances for different workloads and to configure sufficient space. Thus making it easier to meet your SLA (Service Level Agreements).

Timeout between FeedMaster and Clients 

The timeout between the FeedMaster and client triggering requests to Ultra Pipeline Task is user configurable. You can configure the timeout on a per-request basis by passing an X-SL-RequestTimeout header with the timeout value stated as the number of milliseconds (the default is 900000). You can also configure the timeout for the FeedMaster as a whole with the jcc.llfeed.request_timeout global property.

Ultra requests are guaranteed to run at least once before the timeout.

See Also

Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.