Expand | ||
---|---|---|
| ||
Overview
...
Overview
This article describes the relationship between pipeline constructions in SnapLogic® and the Control Plane and Data Plane. Although nodes communicate within a Snaplex, Snaplexes do not directly communicate with each other, regardless of whether the Snaplex is managed by SnapLogic (Cloudplexes) or self-managed (Groundplexes). Because Snaplexes only communicate through the control plane, understanding the dependencies between the control plane and a Groundplex is helpful for implementation planning, pipeline design considerations, and troubleshooting. Pipelines can
Guidelines for Reducing Dependency on the Control Plane
Under certain conditions, pipelines can continue to process documents for a while even if the some hours when a Groundplex node cannot communicate with the control plane.
Recommendations
To build pipelines that However, to reduce dependency on the control plane further, we recommend the following practices when building pipelines:
Avoid using SLFS files (references to files uploaded to Manager and referenced in pipelines)
Avoid files directly stored files on customer self-managed cloud storage like Amazon S3.
Avoid using dynamic references for assets like such as accounts; instead, use static references. Statically referenced assets (such as accounts and child pipelines) are cached on the Snaplex.
The following types of pipelines run without access to the control plane using the aforementioned recommendationsthese guidelines:
Processing Pipelines that process documents through an Ultra pipeline taskPipeline Task, which is already running, using the Low-latency Feed (Request and Response) design.
Pipelines, which Standard-mode pipelines that are already started , either a standard mode pipeline or an Ultra pipeline task using Pipeline Tasks that use the Always-On (Continuous Data Flow) design.
Dependencies between Groundplex and Control Plane
The following table lists the dependencies between a Grounplex Groundplex and the control plane for pipelines invoked as Ultra pipeline tasks, triggered tasks, scheduled tasksPipeline Tasks, Triggered Tasks, Scheduled Tasks, and child pipelines.
Tyoe Type of Integration | Asset | Dependency (Yes/No) | NotesDetails |
---|---|---|---|
A running Ultra Task with No without Child Pipelines - Running | Accounts | No |
|
Expression Library | No |
| |
Pipelines | No |
| |
SLDB File Read from Manager | Yes | Do not use SLDB files in Manager as file source. | |
SLDB File Write from Manager | Yes | Do not use SLDB as file destination. | |
Metadata Snap Pack | Yes | None | |
Ultra Task with No Child Pipelines without child pipelines - Initializing | Accounts | Yes |
|
Expression Library | Yes |
| |
Pipelines | Yes |
| |
SLDB File Read | Yes | Do not use SLDB files in Manager as file source. | |
SLDB File Write | Yes | Do not use SLDB as file destination. | |
Metadata Snap Pack | Yes | None | |
Ultra Task with child pipelines - initialization state or already running with child pipelines not running yet | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Scheduled Task (Initializing) Triggered Task - Cloud URL | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
(Control Plane) Triggered Task - Alternative or Snaplex URL | Accounts | Yes | None |
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Scheduled Task (Already Running) | Accounts | No | Assumes the task is running for a very long time (such as days or weeks). Sample use case: Salesforce Subscriber Snap always running, or a File Poller snap Snap that polls indefinitely. However, you would not be able to cannot view Pipeline Execution statistics in the dashboard if the control plane goes down. |
Expression Library | No | ||
Pipelines | No | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes | ||
Scheduled Task (Initializing) | Accounts | Yes | None|
Expression Library | Yes | ||
Pipelines | Yes | ||
SLDB File Read | Yes | ||
SLDB File Write | Yes | ||
Metadata Snap Pack | Yes |
Troubleshooting
Scenario | Work-around |
---|---|
Most Scheduled Tasks and Ultra Pipeline Tasks when it’s the pipeline is in the prepare state happen on the Snaplex. However, scheduled tasks still have a dependency on the control plane. | Change to an Ultra task Task pipeline. |
Statically defined OAuth accounts depend on Control Plane the control plane - specifically the refresh token. | ou You can use client credentials in-memory to get around this which would depend on the Snaplex, instead of the Control Plane. |