Using Long-running Triggered Tasks with an Enterprise Scheduler
In this article:
Overview
Enterprise job schedulers that execute long-running Triggered Task Pipelines can report timeout or lost client connection errors. To help solve this issue, we provide the snaplogic_exec.sh
script. It can execute on any server that has network access to call the Triggered Task URL.
Scheduler errors occur when network configuration limits the idle time for REST calls and a Triggered Task does not finish within that limit. For example, many network configurations limit the idle time for REST calls to one hour. With long-running Pipelines, such as for overnight batch processing, the synchronous call invoking the Triggered Task might not return within an hour. This can cause the scheduler to report errors even if the Pipeline is still running and eventually completes successfully.
Instead of invoking the Triggered Task directly, you can set up a scheduler to execute the Pipeline from the script. The script starts Pipeline execution and polls continuously for completion status. This keeps the connection alive and prevents the scheduler from reporting an error.
Prerequisites
A user or service account with read and execute access for the project that contains the Triggered Task.
The URL for the Triggered Task, including any Pipeline parameters and their values.
Read and execute permissions on the node(s) that will run the Triggered Task.
Workflow
Install the Script
Install the script on any server where the Scheduler has access to it and configure it with your SnapLogic credentials:
Download snaplogic_exec.zip.
Extract the compressed file and save the
snaplogic_exec.sh
script to the appropriate location.
For example, use/usr/local/bin
on Linux machines and%systemroot%\System32\Repl\Imports\Scripts
on Windows.From the directory containing the script, invoke
snaplogic_exec.sh --config
in a terminal window. For example, on Windows:Enter the SnapLogic user name and password.
Next, configure the Pipeline to work with the script.
Configure the Pipeline
To configure the Pipeline to work with the script, add a Mapper Snap that returns the Runtime ID (RUUID) value when the Triggered Task runs. The script will monitor execution using the RUUID.
In Snaplogic Designer, open the Pipeline that you want the script to execute.
Add a Mapper Snap to the canvas.
In the Settings tab, enter Map snaplogic_exec.sh return in the Label field.
In the Mapping table, add the Expression, pipe.ruuid with the Target path ruuid:
Select the Views tab.
Click - (minus) to remove the input view.
Leave the default output view,
output0
, as is, with Typedocument
:Save and close the Mapper Snap.
The Pipeline should look similar to the following (before validation):
Next, set up the scheduler to run the script.
Run the script from the scheduler
After installing the script and configuring the Pipeline, invoke the script from the scheduler. The invocation must include your Org name and the Triggered Task URL. For example:
snaplogic_exec.sh --org <my_org> "https://elastic.snaplogic.com/api/1/rest/slsched/feed/<task_path>?param1=<param-value>?param2=<param2-value>"
The following table lists snaplogic_exec.sh
command options.
Command | Shortcut | Description |
---|---|---|
|
| Creates or updates a |
|
| Specifies the SnapLogic Org containing the Pipeline to execute. (Required) |
|
| Specifies the number of seconds between Public Monitoring API calls. (Default 30) |
|
| Specifies the maximum number of times to retry the the Public Monitoring API call before terminating the script |
|
| Enables more messages during script execution |
|
| Enables debug messages during script execution |
| Â | Outputs parameters and command options to the terminal window |
Using the snaplogic_exec.sh
script should enable you to execute long-running Triggered Tasks from an enterprise scheduler without causing timeout or lost connection errors.
Troubleshooting
If you encounter issues with the script, run it using the --debug
and --verbose
commands. They display messages as commands execute, and report all responses.
Related Information
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.