You can invoke a Pipeline execution through Tasks in the SnapLogic Platform. Tasks are how Pipelines become operational. As a general example, consider the Pipeline designer as a developer, typically with the background of an IT specialist, who builds Pipelines that fit the data processing needs of an organization. After you design a Pipeline, you then need to run it on a schedule or after an event occurs. Hence, Tasks provide an easy way to accomplish the productionization of your Pipelines.
Tasks enable you to execute your Pipelines by either using a schedule or by triggering a URL.
Scheduled Tasks: Choose this option if you need to accomplish a job at a certain time, an interval, or a more complex schedule.
Triggered Tasks: Choose this option to enable triggering the Pipeline execution through an HTTP call. A Triggered Task can be used to build an endpoint of a web API. A Triggered Task also allows passing data into and retrieving data from a Pipeline.
Ultra Pipeline Tasks: Choose this option for either specialized, low-latency jobs that need to process documents continuously or for Pipelines based on the always-on design. For the former, a FeedMaster node is required in the Snaplex to queue the incoming messages. The URL method is similar to Triggered Tasks, but the Pipeline design limits the usage of certain Snaps. For the latter, the Pipeline can continuously poll the target messaging service, making the Ultra Task preferable to a Scheduled Task.
The difference between Ultra and Triggered Tasks is that the Ultra Task is a constantly running Pipeline, listening for the new documents streaming in. By the time a document is sent to an Ultra Task, the underlying Pipeline is already prepared and can start processing the document instantly, whereas a Pipeline run through the URL that is created from the Triggered Task has to go through the Pipeline preparestage first. Depending on a variety of criteria (Pipeline size, accounts, and types of Snaps used in the Pipeline), the prepare stage can take time, which makes the Ultra Task usage beneficial when the expected response time is a matter of sub-seconds. Since Ultra Pipelines are always running, they can be used to process documents continually from external sources like message queues. Also, data passed into an Ultra pipeline is more reliably processed.
In terms of Pipeline design, the Ultra Task is more restricting when compared to Triggered Tasks because of the number of unsupported Snaps and restrictions around the input and output Snaps and Pipeline parameters. In addition, the Snaplex on which the Ultra Task runs must have a FeedMaster.
Triggered and Ultra Tasks have the following four authentication mechanisms:
Basic Auth using Service Accounts: The easiest method to use, this type of authentication is shared across Tasks based on user ACLs. It supports password expiration.
Bearer token: Also easy to use, the bearer token is by default unique per Task—one token per task. The token can be manually synced across tasks.
JWT-based access control: This method suits Ultra Tasks. The authentication can be implemented in the Pipeline using JWT Snaps. This method allows for more fine-grained access control, at the project level or based on other criteria. JWT also allows for multiple tokens per Task.
You can use the service account and bearer token for simple API integration use cases, and forcomplex use cases, you can use JWT and API Policy Manager.
You can configure event-based notifications for your Tasks. In the dialog windows of all three Task types, you can add or remove recipients and set the event type for the notification. For example, a Task owner might enter the Org admin as a recipient and select Stopped as the event type, which indicates that the Task execution did not complete.
Both Email and Slack are supported for Task Notifications. For Slack recipients, enter the Slack Channel or direct username.