Managing Snaplex Instances


Overview

A Snaplex is the data processing engine of the SnapLogic Intelligent Integration Platform (IIP). You can deploy one or many Snaplex instances as required to run Pipelines and process data. A Snaplex consists of one or more nodes and can be one of the following types:

  • Cloudplex: All Cloudplex instances run inside the SnapLogic Integration Cloud. Use SnapLogic Manager and Dashboard to administer and monitor your Cloudplex. The SnapLogic DevOps team administers the infrastructure key performance indicators (KPI). Cloudplex is ideal if you require integrations that orchestrate across cloud applications (such as Salesforce, ServiceNow, Workday) with no on-premises connections that do not require any software to run behind a firewall.
  • Groundplex: If you need on-premises connectivity (such as SAP, Oracle, Microsoft Dynamics AX) then you require a Groundplex that runs behind the firewall. Although Groundplex nodes run on private or virtual private data centers, Groundplex instances are managed remotely by the SnapLogic Platform's control plane. 
  • eXtremeplex: The Snaplex type for use in a Big Data environment to process data at scale in cloud infrastructure such as AWS EMR or Azure Databricks. See SnapLogic eXtreme for more information. 

Selecting a Snaplex for Pipelines

You can configure which Snaplex a Pipeline will use in the following ways:

  • With a Pipeline open in Designer, you can select from the Snaplex drop-down menu a Snaplex instance on which the Pipeline should run.
  • When you configure a Scheduled or Triggered Task, you can select the Snaplex to run the Pipeline.
  • The Pipeline Execute Snap has a Snaplex property that specifies where the child Pipeline is executed. When using a Pipeline Execute Snap, if you leave the property blank or use pipe.plexPath expression, the child Pipeline is executed on the same node as the parent. When running a parent and child Pipeline on the same node, data is not transferred over a network, which improves the performance of the Pipeline execution.
  • When using the Pipeline Execute Snap, if you specify a different Snaplex to run a child Pipeline, then the input documents and Pipeline parameters are passed (through the control plane using encrypted transport) to a node in the designated Snaplex.

  • To set a dedicated Snaplex for a project, an Org admin can move a Snaplex from the Shared project to any other project. Only those Pipelines within the new project have access to that Snaplex. 

Snaplex State

In the default state, all nodes in the Snaplex can accept new Pipeline execution requests, and Pipeline execution is load-balanced automatically across the nodes.

Snaplex State Transition

A state transition can occur for many reasons–because of a Snaplex version update, a configuration change to the Snaplex, or a restart. When a state transition is required for a Snaplex, one node (or more for larger Snaplex instances) at a time enters a Cooldown state, in which new Pipeline execution requests are not sent to the node. In this state, the node waits for currently running Pipelines to complete. For long-running Pipelines that are set up to poll continuously (like a Pipeline using File Poller or JMS Consumer Snaps) and Ultra Pipelines, the Pipeline is sent a state notification to allow the Pipeline to stop polling for new records.

Pipeline Execution Timeout During Restart

A node can stay in the Cooldown state for a configurable timeout (the default is 15 minutes), called the Restart Max Wait Time, which you can set in the Snaplex Settings tab in the Update Snaplex and Create Snaplex dialogs. If all Pipelines complete within this timeout, then the node transitions to the required state. If Pipelines are still running after this timeout, then those Pipelines terminate to allow the transition to the required state.

You can increase the default value of 15 minutes to a higher value on Snaplex instances that run longer-running Pipelines (in terms of duration). However, for polling and Ultra Pipelines, you do not have to increase this value since the Pipeline is notified to stop polling for new records, and thus allowing the Pipeline to terminate cleanly.

Guidelines for Setting the Restart Max Wait Timeout

  • We strongly recommend that you keep the Restart Max Wait Time below one hour. Setting a larger timeout can cause issues because configuration and version changes take a longer time to apply. During the final Snaplex version upgrade (five weeks after the release), the SnapLogic control plane waits one hour for all Snaplex instances still on the older version to upgrade to the new version. Beyond one hour, the control plane does not allow new Pipeline execution requests on the old version.
  • If you use the Snaplex for Triggered Pipeline or Ultra Pipeline executions, we strongly recommend you to setup a load balancer on the Snaplex configured to use a health check on the node state. Without a properly configured load balance, requests would continue to come to the node while in Cooldown state, causing failures.
  • You can set the timeout to Forever, which would never allow the Snaplex to go into a transition state. However, setting the timeout to Forever places the onus of manual termination of Pipelines on the Org admin. Essentially, you would need to terminate some Pipelines manually to force a state transition in the Snaplex.
  • If you decrease the timeout to an extremely low value, then any Pipeline of a longer duration is terminated before completion.

Snaplex Version Change

When a Snaplex version change is initiated, one node at a time undergoes a transition of the state. After the node enters the Cooldown state and running Pipelines complete, the JCC process on the node terminates and restarts with the new version of the Snaplex binaries.
 Since the Snaplex version upgrade happens in a rolling manner with no downtime required on Pipeline executions, your interaction is not required during the rolling restart—all node transitions are completed automatically.

Snaplex Maintenance Mode


When you place a Snaplex node in maintenance mode, the node enters the Cooldown state and waits for running Pipelines to complete. After all Pipelines complete, the node enters maintenance mode, during which the node does not accept any Pipeline execution requests.

During an upgrade, the behavior of the Snaplex depends on the type:

Cloudplex: If a node is in maintenance mode and the Snaplex version is updated, the Cloudplex exits maintenance mode after the version update.

Groundplex: If a node is in maintenance mode and the Snaplex version is updated, the Groundplex remains in the maintenance mode after the update.

  • We do not recommend manually placing the nodes in maintenance mode during a Snaplex version update or configuration change.
  • The FeedMaster should not receive client requests while it is in maintenance mode. If you put a FeedMaster node in maintenance mode, you must make sure your load balancer is set up to stop client requests from directing to the FeedMaster when the health check to the node is failing. The health check fails when the node is in maintenance mode.

Cloudplex Behavior 

Cloudplex nodes reinitialize with each Snaplex version upgrade. Subsequently, you will notice that any files created on the local file system (using file:// protocol) will not be accessible across the Snaplex upgrades, since the nodes are new. Hence, we recommend using Cloud storage for files you need to access across the Snaplex upgrade process. SnapLogic Platform does not perform a backup of files created on the Cloudplex node's local file system.