Overview
Snaplex nodes can cache Pipelines for up to two hours when communication with the control plane is unavailable. Although Snaplex nodes communicate within a Snaplex, Snaplexes do not directly communicate with each other, regardless of whether the Snaplex is managed by SnapLogic (Cloudplexes) or self-managed (Groundplexes). Because Snaplexes only communicate through the control plane, understanding the dependencies between the control plane and a Groundplex is helpful for implementation planning, pipeline design considerations, and troubleshooting.
Recommendations
To build pipelines that reduce dependency on the control plane:
Avoid using SLFS files (files uploaded to Manager and referenced in pipelines)
Avoid directly stored files on customer self-managed cloud storage like S3.
Avoid using dynamic references for assets like accounts; instead, use static references. Statically referenced assets (such as accounts and child pipelines) are cached on the Snaplex
The following types of pipelines run without access to the control plane using the aforementioned recommendations:
Processing documents through an Ultra pipeline task, which is already running, using the Low-latency Feed (Request and Response) design.
Pipelines, which are already started, either a standard mode pipeline or an Ultra pipeline task using the Always-On (Continuous Data Flow) design.
Secure Connectivity
All account credentials are stored in an AWS S3 bucket and encrypted with RSA encryption using key sizes of 4096 bits. Pipeline details are confidential and stored in Mongo DB. Sensitive mappings should be put in an expression library which will be stored encrypted on S3. MongoDB respects firewall rules, and only authorized users can connect to it.
Snaplogic does not store or persist customer data. Metadata is secured at the network, Org level, and DB authorization and access controls. Data is isolated to the customer instance, and metadata is logically separated in the database. The Control Plane does not initiate contact with the Groundplex. MongoDB access is further restricted to service accounts within the application.
For additional controls, Enhanced Account Encryption allows Groundplex users to manage and maintain the private keys.
Dependencies between Groundplex and Control Plane
The following table lists the dependencies between a Grounplex and the control plane for pipelines invoked as Ultra pipeline tasks, triggered tasks, scheduled tasks, and child pipelines.
Tyoe of Integration | Asset | Dependency (Yes/No) | Notes |
|---|---|---|---|
Ultra Task with No Child Pipelines - Running | Accounts | No |
|
Expression Library | No |
| |
Pipelines | No |
| |
SLDB File Read | Yes | Do not use SLDB as file source. | |
SLDB File Write | Yes | Do not use SLDB as file destination. | |
Metadata Snap Pack | Yes | None | |
Ultra Task with No Child Pipelines - Initializing | Accounts | Yes |
|
Expression Library | Yes |
| |
Pipelines | Yes |
| |
SLDB File Read | Yes | Do not use SLDB as file source. | |
SLDB File Write | Yes | Do not use SLDB as file destination. | |
Metadata Snap Pack | Yes | None | |
Ultra Task with child pipelines - initialization state or already running with child pipelines not running yet | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Triggered Task - Cloud URL | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Triggered Task - Alternative or Snaplex URL | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Scheduled Task (Already Running) | Accounts | No | Assumes the task is running for a very long time (such as days or weeks). Sample use case: Salesforce Subscriber Snap always running, or a File Poller snap that polls indefinitely. However, you would not be able to view Pipeline Execution statistics in the dashboard if the control plane goes down. |
Expression Library | No | ||
Pipelines | No | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Scheduled Task (Initializing) | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes |
Troubleshooting
Scenario | Work-around |
|---|---|
Most Scheduled Tasks and Ultra when it’s in the prepare state happen on the Snaplex. However, scheduled tasks still have a dependency on the control plane. | Change to an Ultra task pipeline. |
Statically defined OAuth accounts depend on Control Plane - specifically the refresh token. | ou can use client credentials in-memory to get around this which would depend on the Snaplex, instead of the Control Plane. |