Creating a Snaplex

In this article

Overview

Snaplex is a data processing engine used for Pipeline execution. A Snaplex contains multiple nodes and can be a Cloudplex or a Groundplex.

Prerequisites

  • Only an Org Administrator can create a Snaplex.
  • The options available for a Snaplex depend on the Snaplex type.
  1. To create a Snaplex, navigate to the target project in SnapLogic Manager.
  2. Perform either of the following actions:
    • Click  and from the dropdown list select Snaplex 
      OR
    • Click the Snaplex tab in the page that displays the assets for that Project.
      The Create Snaplex dialog displays.
  3. Click each tab to configure the Snaplex: 
  4. Click Create.

Support for Notifications through Slack

SnapLogic supports the Slack messaging app within the SnapLogic platform communications, enabling you as an Org admin to add Slack channels and recipients for your SnapLogic communications. If Slack is enabled in Manager > Settings, the Snaplex Settings tab contains fields to add channels and users to receive notifications. Learn more about adding Slack communications to your Org, SnapLogic Notifications through Slack.

Configuring the Snaplex Settings

The Create Snaplex dialog contains the following tabs:

  • Settings
  • Logging
  • Node Properties
  • Node Proxies (enabled only for Groundplex nodes).

Settings Tab

Name

Specify a unique name for the Snaplex. The name must not exceed 100 characters.

We recommend you use a descriptive name for your Snaplex, that you can identify it easily.

Default Value: None

Location

Select the location where the Snaplex must be created. The available options are:

  • Cloud: Creates a Snaplex in the cloud.
  • Sidekick: Creates a Snaplex on-premise (Groundplex).

Default Value: Sidekick

Environment

Specify the value to configure the Snaplex nodes. Alphanumeric characters and the following special characters are allowed: ( - +)

Default Value: None
Example: Dev-10

You cannot update the environment value if there are nodes running in the Snaplex. You must stop the Snaplex nodes first before updating the environment value.
Download the config file for each node and restart the nodes for the environment value to take effect.

Version

Specify the version of the SnapLogic Platform on which you want the Snaplex to run.

By default, all new Snaplexes are configured to work with the latest version of the SnapLogic platform. To update your Snaplex version, from the Version list, select the SnapLogic Platform version (such as mrc216) that you want to use with the Snaplex. Once you save the Snaplex, a rolling restart of the nodes in the Snaplex is initiated. Sequentially each node in the Snaplex is put offline, and no new Pipelines are sent to the node. The node waits for a maximum of 15 minutes for currently running Pipelines to complete their execution. It then restarts itself with the specified version of the Snaplex binaries. Once these restarts are done, the Dashboard displays the updated nodes to be running with the specified version.

Default value: Default

Email address for notifications

List the email addresses to notify if one of the Snaplex nodes does not respond for fifteen minutes.

Default Value: N/A
Example: testuser@snaplogic.com

Slack Channel for notifications

Specify the name of Slack channels (separated by commas) to notify if one of the Snaplex nodes does not respond in fifteen minutes.

Default Value: None
Example: DevOps

Slack user for notifications

Enter the Slack recipients (separated by commas) to notify if one of the Snaplex nodes does not respond for fifteen minutes.

Default Value: N/A
Example: testuser

Load balancer

Specify the URL for the load balancer for Triggered Task execution requests. The load balancer URL has to be fully qualified, including the protocol. 

When you save the Load balancer URLs in the Snaplex settings, these details are used for Snaplex Trigger URL.

Default Value: None
Example: https://snaplexlb.mydomain.com

Ultra load balancer

Specify the URL of the FeedMaster load balancer for Ultra Pipeline execution requests.

This property is available only to Orgs that have subscribed to Ultra Pipeline Tasks.

Default Value: None
Example: https://ultralb.mydomain.com

Snaplex Configuration changes

Snaplex nodes are typically configured using a slpropz configuration file, located in the $SL_ROOT/etc folder

If you use the slpropz file as your Snaplex configuration, then:

  • After a Snaplex node is started with the slpropz configuration, subsequent configuration updates are applied automatically.
  • Changing the Snaplex properties in Manager causes each Snaplex node to download the updated slpropz and do a rolling restart with no downtime on Snaplex instances with more than one node.
  • Some configuration changes, such as an update to the logging properties do not require a restart and are applied immediately.
  • Some configuration properties like the Environment cannot be changed without doing manual updates for the slpropz files on the Snaplex nodes.

Older Installations

If you have an older Snaplex installation and its configuration is defined in the global.properties file, then the Environment value must match the jcc.environment value In the JCC global.properties file. To migrate your Snaplex configuration to the slpropz mechanism, see Migrating Older Snaplex Nodes.

You should always configure your Snaplex instances using the slpropz file because you do not have to edit the configuration files manually. Changes to the Snaplex done through Manager are applied automatically to all nodes in that Snaplex.

Load Balancer Settings

For Snaplex triggered pipeline and for Ultra API calls, a load balancer needs to be configured. SnapLogic will provision the load balancer for Cloudplexes when the Ultra feature is enabled. For Groundplexes, the customer has to provision the load balancer. After the load balancer is provisioned, the load balancer URL needs to be configured in the Snaplex properties. This will ensure that task details show the load balancer URLs rather than pointing to the Snaplex nodes.

For Snaplexes that have FeedMasters, the load balancer should point to the FeedMaster nodes. The "Ultra load balancer" property should be updated to point to the load balancer URL. This will ensure that all tasks details will point to this load balancer. This includes ultra and triggered tasks.

For Snaplexes that do not have FeedMasters, the load balancer should point to the JCC nodes. The "Load balancer" property should be updated to point to the JCC load balancer URL. The triggered tasks details page will point to this load balancer.

The load balancer is recommended to be configured with HTTPS enabled. If the load balancer URL is configured with HTTP instead of HTTPS, then the task details page will point to the load balancer for the HTTP URLs only. The HTTPS URLs will point to the Snaplex nodes rather than pointing to the load balancer.

Check Groundplex Requirements#LoadBalancerGuidelines for more details about the load balancer configuration guidelines.

Logging Tab

Click the Logging tab to configure logs associated with the Snaplex that you want to create. 

Note that the load balancer details are not populated for triggered tasks though provided in the Snaplex settings.


Level

Specify the minimum level of logging–or the kind of details–that you want to enable for the new Snaplex.

Available values:

  • Trace: Records details of all events associated with the Snaplex.
  • Debug: Records all events associated with the Snaplex.
  • Info: Records messages that outline the status of the Snaplex and the tasks completed.
  • Warning: Records all warning messages associated with the Snaplex.
  • Error: Records all error messages associated with the Snaplex.

Default Value: Debug

Log file size

The maximum size of the log file to be created for the Snaplex.

Default Value: 50 MB

We recommend that you use a descriptive name for your Snaplex, so that you can identify it easily.

Main backup count

The number of backup main log files that SnapLogic must maintain for the Snaplex.

Default Value: 10

Error backup count

The number of backup error log files that SnapLogic must maintain for the Snaplex.

Default Value: 5

Access backup count

The number of backup access log files that SnapLogic must maintain for the Snaplex.

Default Value: 5

Node Properties Tab

Click the Node Properties tab to configure the nodes associated with the Snaplex that you want to create.

Maximum slots

Each Snap in a Pipeline consumes a slot; so Pipelines can only be executed on nodes where the number of slots in use is below this threshold. Otherwise, they will fail or queued, depending on how they were executed. The number of slots in use corresponds to the number of active threads on a node, which can be viewed in the Snaplex Health dashboard.

Default Value: 4000

A heuristic to use is that 2000 slots can be allocated per 8GB of RAM available on each Snaplex node. So for nodes with 16GB of RAM, use 4000 as the slot count.

Reserved slots are used when a Pipeline or validation is executed manually from SnapLogic Designer. If no slots are reserved, then these interactive Pipeline executions—Pipeline executions run from Designer for testing or validation purposes—can be blocked by long running executions. You can adjust this setting based on your experience if the default value is not optimal. To reserve more slots for interactive sessions, set your value lower. If the Snaplex is not being used for interactive Pipeline building, set this value higher.

Reserved slot %

The percentage of slots that you want to reserve on a node for Pipelines executed through the Designer tab. Pipelines executed using Tasks or the ForEach Snap will not have access to these slots.

Changes made to this setting do not require a restart.

Default Value: 15

Maximum memory %

The memory threshold at–and above–which no more Pipelines can be assigned to a node.

Changes made to this setting do not require a restart.

Default Value: 85

Heap max size

Groundplex only. The maximum JVM heap size.

Default Value: auto (meaning that SnapLogic will automatically set the max heap size based on the available machine memory).

HTTP Interface

Groundplex only. Specify the location from which the Snaplex node can accept HTTP network connections. The following options are available:

  • Localhost only (127.0.0.1)
  • Any interface (0.0.0.0)

Default Value: Localhost only (127.0.0.1)

HTTP Port

Groundplex only. The HTTP port on which the Snaplex node listens for connections.

Default Values: 8090 for a JCC node and 8091 for a FeedMaster.

HTTPS Port

Groundplex only. The HTTPS port on which the Snaplex node listens on for connections. 

Default Values: 8081 for a JCC node and 8084 for a FeedMaster.

Restart Maximum Wait Time

The maximum wait time before restarting a node.
Enter the maximum wait time or click Forever for an infinite wait time. When a Snaplex node goes into Maintenance mode or a Cooldown state, any Pipeline executions exceeding this value will cause those Pipelines to fail.
Learn more about the Snaplex states.

Default Value: 15 minutes

Global Properties (Key/Value pairs)

Groundplex only. Internal configuration options. Do not edit these values without contacting SnapLogic Customer Support.

Default Value: None


If you receive the following error: "reach the innerElementCountThreshold:50000", perform one of the following tasks:

  • Configure the Global Properties of the Groundplex in the global.properties file ("/opt/snaplogic/etc/global.properties" in Linux; "c:\opt\snaplogic\etc\global.properties" in Windows) to increase the threshold value:
    jcc.jvm_options = -Dorg.apache.cxf.stax.maxChildElements=<value>
    where 'value' is 1000000 or higher. OR
  • Add the same line as a global property in the node.

These properties are available only in a Groundplex.


Node Proxies Tab

Use the Node Proxies tab to control how the nodes communicate with an HTTP/HTTPS proxy server to communicate with the outside world. The Groundplex nodes must have been setup to use an .slpropz configuration file before changes to these properties will take effect. If you make changes that affect the software configuration, but there are nodes in the Snaplex that are not setup to use a .slpropz configuration file, a warning dialog will appear with a listing of the unmanaged nodes. See the Downloads section in Updating a Snaplex for more information on setting up a node to use this configuration mechanism.

HTTP proxyConfiguration details associated with the HTTP proxy server.
Hostname

The URL of the HTTP proxy host.

Default Value: N/A.
Example: 172.0.10.162

Port

The port number on which the HTTP proxy host listens.

Default Value: N/A
Example: 3127

Non-proxy hosts: Host patternThe hostnames or IP addresses that should be contacted directly instead of through the proxy. Patterns may start or end with a * for wildcards.
HTTPS proxyConfiguration details associated with the HTTPS proxy server.
Hostname

The URL of the HTTPS proxy host.

Default Value: No default value.
Example: 172.0.10.162

Port

The port number on which the HTTPS proxy host listens.

Default Value: N/A
Example: 3127

Non-proxy hosts: Host patternThe hostnames or IP addresses that should be contacted directly instead of through the proxy. Patterns may start or end with a * for wildcards.