Overview
A Snaplex is the data processing engine of the SnapLogic Intelligent Integration Platform (IIP). You can deploy one or many Snaplex instances as required to run Pipelines and process data. A Snaplex consists of one or more nodes and can be one of the following types:
- Cloudplex: All Cloudplex instances run inside the SnapLogic Integration Cloud. Use SnapLogic Manager and Dashboard to administer and monitor your Cloudplex. The SnapLogic DevOps team administers the infrastructure key performance indicators (KPI). Cloudplex is ideal if you require integrations that orchestrate across cloud applications (such as Salesforce, ServiceNow, Workday) with no on-premises connections that do not require any software to run behind a firewall.
- Groundplex: If you need on-premises connectivity (such as SAP, Oracle, Microsoft Dynamics AX) then you require a Groundplex that runs behind the firewall. Although Groundplex nodes run on private or virtual private data centers, Groundplex instances are managed remotely by the SnapLogic Platform's control plane.
- eXtremeplex: The Snaplex type for use in a Big Data environment to process data at scale in cloud infrastructure such as AWS EMR or Azure Databricks. See SnapLogic eXtreme for more information.
Selecting a Snaplex for Pipelines
You can configure which Snaplex a Pipeline will use in the following ways:
- With a Pipeline open in Designer, you can select from the Snaplex drop-down menu a Snaplex instance on which the Pipeline should run.
- When you configure a Scheduled or Triggered Task, you can select the Snaplex to run the Pipeline.
- The Pipeline Execute Snap has a Snaplex property that specifies where the child Pipeline is executed. When using a Pipeline Execute Snap, if you leave the property blank or use pipe.plexPath expression, the child Pipeline is executed on the same node as the parent. When running a parent and child Pipeline on the same node, data is not transferred over a network, which improves the performance of the Pipeline execution.
When using the Pipeline Execute Snap, if you specify a different Snaplex to run a child Pipeline, then the input documents and Pipeline parameters are passed (through the control plane using encrypted transport) to a node in the designated Snaplex.
- To set a dedicated Snaplex for a project, an Org admin can move a Snaplex from the Shared project to any other project. Only those Pipelines within the new project have access to that Snaplex.
Snaplex State
In the default state, all nodes in the Snaplex can accept new Pipeline execution requests, and Pipeline execution is load-balanced automatically across the nodes.
Snaplex State Transition
A state transition can occur for many reasons–because of a Snaplex version update, a configuration change to the Snaplex, or a restart. When a state transition is required for a Snaplex, one node (or more for larger Snaplex instances) at a time enters a Cooldown state, in which new Pipeline execution requests are not sent to the node. In this state, the node waits for currently running Pipelines to complete. For long-running Pipelines that are set up to poll continuously (like a Pipeline using File Poller or JMS Consumer Snaps) and Ultra Pipelines, the Pipeline is sent a state notification to allow the Pipeline to stop polling for new records.
Pipeline Execution Timeout During Restart
A node can stay in the Cooldown state for a configurable timeout (the default is 15 minutes), called the Restart Max Wait Time, which you can set in the Snaplex Settings tab in the Update Snaplex and Create Snaplex dialogs. If all Pipelines complete within this timeout, then the node transitions to the required state. If Pipelines are still running after this timeout, then those Pipelines terminate to allow the transition to the required state.
You can increase the default value of 15 minutes to a higher value on Snaplex instances that run longer-running Pipelines (in terms of duration). However, for polling and Ultra Pipelines, you do not have to increase this value since the Pipeline is notified to stop polling for new records, and thus allowing the Pipeline to terminate cleanly.
Note | ||
---|---|---|
| ||
|
Snaplex Version Change
When a Snaplex version change is initiated, one node at a time undergoes a transition of the state. After the node enters the Cooldown state and running Pipelines complete, the JCC process on the node terminates and restarts with the new version of the Snaplex binaries. Since the Snaplex version upgrade happens in a rolling manner with no downtime required on Pipeline executions, your interaction is not required during the rolling restart—all node transitions are completed automatically.
Snaplex Maintenance Mode
When you place a Snaplex node in maintenance mode, the node enters the Cooldown state and waits for running Pipelines to complete. After all Pipelines complete, the node enters maintenance mode, during which the node does not accept any Pipeline execution requests.
During an upgrade, the behavior of the Snaplex depends on the type:
Cloudplex: If a node is in maintenance mode and the Snaplex version is updated, the Cloudplex exits maintenance mode after the version update.
Groundplex: If a node is in maintenance mode and the Snaplex version is updated, the Groundplex remains in the maintenance mode after the update.
Note |
---|
|
Cloudplex Behavior
Cloudplex nodes reinitialize with each Snaplex version upgrade. Subsequently, you will notice that any files created on the local file system (using file:// protocol) will not be accessible across the Snaplex upgrades, since the nodes are new. Hence, we recommend using Cloud storage for files you need to access across the Snaplex upgrade process. SnapLogic Platform does not perform a backup of files created on the Cloudplex node's local file system.