Pipeline Monitoring API
This topic might be out-of-date and is replaced by the following:
In this article
Overview
The following SnapLogic APIs enable you to programmatically query Pipeline runtimes and collect API metrics for your Triggered and Ultra Tasks. This article explains the API calls, parameters, and examples.
Monitor Pipelines
Getting Pipeline Runtimes for your Org
Endpoint
GET /api/1/rest/public/runtime/<org>?
Description
Returns Pipeline runtime data for the specified organization for a given period of time. If a timespan is not set, this API returns information for the last hour.
Request Parameters
Parameter Name | Data Type | Description | Default Value | Example |
---|---|---|---|---|
org | String | Required. The name of the Org. | N/A | DevOrg |
start | Integer | The start timestamp in milliseconds. | N/A | 1426809189000 |
end | Integer | The start timestamp in milliseconds. | N/A | 1426852861000 |
offset | Integer | The start point of the records. | 0 | 0 |
limit | Integer | The maximum number of runtimes to return in a query. The maximum possible is 1000. | 10 | 25 |
last_hours | Integer | Returns data for last n hours. The maximum limit is 1080 hours (45 days). | 1 | 48 |
pipe_name | String | Returns data for a particular Pipeline. | N/A | test pipeline |
org_wide | Boolean | Returns data for whole org including all phases. | N/A | global prod |
state | String | Indicates one of the following Pipeline states.
| N/A | Started |
invoker | String | Returns data for users who invoke the API call with this parameter. Invoker. The type of triggering mechanism for the Pipeline (case sensitive)
| N/A | invoker=scheduled |
plex_label | String | Returns the runtime information for a particular Snaplex label. | N/A | east plex |
cc_label | String | Returns the information for a particular cc label (JCC node’s hostname). | N/A | node 2 |
user_id | String | Returns information for particular users. | N/A | user_id=user_a@example.com,user_b@example.com |
project_path | String | Returns the runtime information for particular project.
| N/A | /Snaplogic/projects/test_project |
mode | String | Indicates the mode of the Pipeline. | N/A | Standard |
Request Example
GET /api/1/rest/public/runtime/snaplogic?offset=0
Response Example
{ "response_map": { "entries": [ { "pipe_id": "29cd6375-b460-4456-8fa7-08a85756394f", "has_lints": false, "documents": 1, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_e929e44c-f513-4fe4-984e-a8a37533cc23", "subpipes": {}, "state_timestamp": "2021-06-11T21:03:45.685000+00:00", "error_documents": 0, "label": "SUB - Generic Parameters for Pipelines", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Completed", "create_time": "2021-06-11T21:03:45.685000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_0daa1437-bc43-46ec-8858-1e95cc1887d2", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "f1988c3f-265a-4168-919e-b67275297b82", "has_lints": false, "documents": 0, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_1af484cb-f380-4941-ad27-d6e6c04c2a0b", "subpipes": {}, "state_timestamp": "2021-06-11T20:36:37.495000+00:00", "error_documents": 0, "label": "Internal - Join and Aggregate", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Failed", "create_time": "2021-06-11T20:36:37.495000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_665d5ce2-7fe3-4789-8dee-ce8a089d2509", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "f1988c3f-265a-4168-919e-b67275297b82", "has_lints": false, "documents": 0, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_1af484cb-f380-4941-ad27-d6e6c04c2a0b", "subpipes": {}, "state_timestamp": "2021-06-11T20:36:37.489000+00:00", "error_documents": 0, "label": "Internal - Join and Aggregate", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Failed", "create_time": "2021-06-11T20:36:37.489000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_b9ed5a7b-e02c-446b-b2c2-8f9214ece448", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "f1988c3f-265a-4168-919e-b67275297b82", "has_lints": false, "documents": 0, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_1af484cb-f380-4941-ad27-d6e6c04c2a0b", "subpipes": {}, "state_timestamp": "2021-06-11T20:36:37.477000+00:00", "error_documents": 0, "label": "Internal - Join and Aggregate", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Failed", "create_time": "2021-06-11T20:36:37.477000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_7b6c36a2-0f44-4839-8d82-19a6ec7548b9", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "f1988c3f-265a-4168-919e-b67275297b82", "has_lints": false, "documents": 0, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_1af484cb-f380-4941-ad27-d6e6c04c2a0b", "subpipes": {}, "state_timestamp": "2021-06-11T20:36:37.453000+00:00", "error_documents": 0, "label": "Internal - Join and Aggregate", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Failed", "create_time": "2021-06-11T20:36:37.453000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_5dd5e8c4-2a57-48ae-baaa-e61d2fca5903", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "29cd6375-b460-4456-8fa7-08a85756394f", "has_lints": false, "documents": 0, "user_id": "schelluri@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": "52e99318640a9a03d8681d0d_1af484cb-f380-4941-ad27-d6e6c04c2a0b", "subpipes": {}, "state_timestamp": "2021-06-11T20:36:37.448000+00:00", "error_documents": 0, "label": "SUB - Generic Parameters for Pipelines", "path_id": "/snaplogic/projects/projects-Production-VerizonProd", "state": "Failed", "create_time": "2021-06-11T20:36:37.448000+00:00", "invoker": "nested_pipeline", "duration": 0, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_0e705bf1-06bd-4d15-bc37-18b73992e120", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "5913896a-6661-4f51-a0c5-d651e3fec3eb", "has_lints": false, "documents": 0, "user_id": "adube@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": null, "subpipes": {}, "state_timestamp": "2021-06-11T21:03:38.225000+00:00", "error_documents": 0, "label": "HTTP Router", "path_id": "/snaplogic/Akhilesh/Snaplogic", "state": "Failed", "create_time": "2021-06-11T21:03:37.225000+00:00", "invoker": "always_on", "duration": 20, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_37a3a034-3096-4297-ba95-cbe9aa7c78a6", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "5913896a-6661-4f51-a0c5-d651e3fec3eb", "has_lints": false, "documents": 0, "user_id": "adube@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": null, "subpipes": {}, "state_timestamp": "2021-06-11T20:58:44.116000+00:00", "error_documents": 0, "label": "HTTP Router", "path_id": "/snaplogic/Akhilesh/Snaplogic", "state": "Failed", "create_time": "2021-06-11T20:58:21.116000+00:00", "invoker": "always_on", "duration": 26, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_e39b48a7-1505-4090-b71e-1cea6f36638e", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "5913896a-6661-4f51-a0c5-d651e3fec3eb", "has_lints": false, "documents": 0, "user_id": "adube@snaplogic.com", "ccid": "60af8acdf0bb32d4cf1a6371", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": null, "subpipes": {}, "state_timestamp": "2021-06-11T20:52:57.956000+00:00", "error_documents": 0, "label": "HTTP Router", "path_id": "/snaplogic/Akhilesh/Snaplogic", "state": "Failed", "create_time": "2021-06-11T20:52:56.956000+00:00", "invoker": "always_on", "duration": 17, "cc_label": "pa22sl-jcc-04c336c2838755d63", "id": "52e99318640a9a03d8681d0d_4b0ac276-3c37-4029-895d-9489172a6471", "runtime_label": "cloud-dev", "mode": "standard" }, { "pipe_id": "5913896a-6661-4f51-a0c5-d651e3fec3eb", "has_lints": false, "documents": 0, "user_id": "adube@snaplogic.com", "ccid": "60976b35ff8ab1a0fa5cfe02", "child_has_lints": false, "runtime_path_id": "snaplogic/rt/cloud/dev", "parent_ruuid": null, "subpipes": {}, "state_timestamp": "2021-06-11T20:47:53.905000+00:00", "error_documents": 0, "label": "HTTP Router", "path_id": "/snaplogic/Akhilesh/Snaplogic", "state": "Failed", "create_time": "2021-06-11T20:47:52.905000+00:00", "invoker": "always_on", "duration": 17, "cc_label": "pa22sl-jcc-0db57ce2fd6d7116b", "id": "52e99318640a9a03d8681d0d_796794c4-ebdb-4adb-82e9-096a71bdef5a", "runtime_label": "cloud-dev", "mode": "standard" } ], "total": 127, "limit": 10, "offset": 0 }, "http_status_code": 200 }
Viewing Information on Specific Pipeline Runtimes
Endpoint
GET /api/1/rest/public/runtime/<Org>/<ruuid>?
Description
Returns Pipeline information for the specified Runtime ID (ruuid).
Parameters
Parameter Name | Data Type | Description | Default Value | Example |
---|---|---|---|---|
org | String | Required. The name of the Org. | N/A | DevOrg |
ruuid | String | The runtime ID for the Pipeline. | N/A | 5b042afda9206611281b7328_abd5b58a-3caf-4b83-bec4-7ffd3ba7fd1u |
level | String | The level of detail: Summary or Detail. | Summary | Detail |
Request Example
GET /api/1/rest/public/runtime/DevOrg/5ed97bd3-8de0-4957-946b-437bd3bc0172
Response Example
{ "response_map":{ "entries":[ { "prepare_handoff_timestamp":"2015-03-20T18:50:04.742000+00:00", "feed_call":false, "class_fqid":"ab3d6766-1848-4509-8916-4393d88a65ce_3", "duration":132, "parent_ruuid":null, "create_time":"2015-03-20T18:50:02.372000+00:00", "snode_id":"54d85e7e4c345d5194e85cfc", "org_snode_id":"52b32bdbf17d831ac1750257", "env_map":null, "instance_version":1, "user_id":"user@snaplogic.com", "project_snode_id":"52b7cf69f17d83721b7276e3", "notification":{ "states":[ "Failed" ], "recipients":"user7@snaplogic.com" }, "label":"Mysql-Redshiftbulkloadtestfeb", "class_version":3, "state":"Completed", "Snaplex_label":"cloud-dev/canxl-jcc1", "time_stamp":"2015-03-20T18:52:15.041000+00:00", "error_list":[ ], "cc_start_ms":1426877404998, "invoker":"user@snaplogic.com", "poll_time_stamp":"2015-03-20T18:52:15.193000+00:00", "ccid":"550c4bbf4c345d21557dd39a", "delivery_options":null, "cc_external_uri":"http://canxl-jccl.snaplogic.com:8090", "runtime_path_id":"Snaplogic/rt/cloud2", "documents_count":100000, "failure":null, "reason":null, "ruuid":"5b042afda9206611281b7328_abd5b58a-3caf-4b83-bec4-7ffd3ba7fd1u", "parent_pipeline_fqid":null, "cc_end_ms":1426877535191, "runtime_label":"cloud-dev", "cc_poll_ms":1426313535190, "class_id":"ab3d6766-1848-011-8916-4393d88a65ce", "statistics_time_stamp":"2015-03-20T18:50:02.372000+00:00", "cc_prepare_start_ms":1426877404757, "instance_id":"5ed97bd3-8de0-4957-946b-437bdy950172", "parent_pipeline_ruuid":null, "plex_path":"/snaplogic/shared/cloud2", "resolution":null, "cc_prepare_end_ms":1426852604907, "pipe_invoker":"scheduled" }, "http_status_code", 200 ] } }
Pipeline Runtime Field Descriptions
Use the following table to understand the field keys in the returned response to the API.
Field Key | Type | Description |
---|---|---|
prepare_handoff_timestamp | String | The Pipeline runtime prepare handoff time. |
feed_call | Boolean | A flag that indicates if the pipeline should increment the API meter (True/False). |
class_fqid | String | The Pipeline class_id with version number at the end. Format: <class_id>_<version>. |
duration | Integer | The Pipeline execution duration |
parent_ruuid | String | The runtime ID of the parent Pipeline. |
create_time | String | The time that the runtime is created. |
snode_id | String | The Snaplex node ID of the Pipeline instance. |
org_snode_id | String | The Snaplex node ID for the Org in which the Pipeline resides. |
env_map | Map | The Snaplex environment. |
instance_version | Integer | The version of instance captured by the API. |
user_id | String | The user who ran the Pipeline. |
project_snode_id | String | The Snaplex node ID for the Project in which the Pipeline resides. |
notification | Map | Indicates if a notification is set up for the Pipeline runtime. |
notification.recipients | String | Indicates who receives the notification. |
notification.states | String | When runtime state is changed to one of the other states, send notification to recipients. |
label | String | The label for the Pipeline. |
class_version | String | The version of the Snap Pack. |
snaplex_label | String | The name of the Snaplex and the JCC node label. |
time_stamp | String | The time stamp of the current operation, regardless of the runtime state. |
error_list | List | List of errors encountered at runtime. |
cc_start_ms | Integer | The start of the runtime on JCC node in milliseconds. |
invoker | String | The user who invoked the API. |
poll_time_stamp | String | The time when the SnapLogic control plane receives the runtime from JCC node. |
ccid | String | The ID for the JCC node instance. |
delivery_options | Map | Delivery options set on a child Pipeline. |
cc_external_uri | String | The external URI of the JCC node executing the Pipeline. |
runtime_path_id | String | The identifier of the Snaplex, made by the following combination: <subscriber_id>/<plex_environment>/<plex_location>. |
documents_count | Integer | The number of documents processed in the runtime. |
failure | String | Indicates if a failure occurred during document processing. |
reason | String | Indicates the reason for a failure. |
parent_pipeline_fqid | String | The unique ID for the Pipeline Execute Snap in the parent Pipeline that called the child Pipeline. |
cc_end_ms | String | The end of the runtime on the JCC Node in milliseconds. |
runtime_label | String | The name of the Snaplex. |
cc_poll_ms | Integer | The time when the JCC node pushes the runtime status in milliseconds. |
class_id | String | Instance identifier of the Pipeline instance. |
statistics_time_stamp | String | The time at which the JCC node collects statistics. |
cc_prepare_start_ms | Integer | The start time when the JCC node prepares the runtime. |
instance_id | String | The Pipeline runtime instance identifier; it is the same as the Pipeline runtime ID. |
parent_pipeline_ruuid | String | The runtime ID of the parent Pipeline. |
plex_path | String | The path of the Snaplex. |
resolution | String | The suggested action to take when the runtime fails. |
cc_prepare_end_ms | Integer | The finish time when the JCC node prepares the runtime. |
pipe_invoker | String | The method of invocation. |
references | List | The references to other SnapLogic Assets in the Pipeline runtime, including the following reference types:
|
message_history | map | The messages from a FeedMaster during the runtime. Each message includes the following timestamps:
|
delivery_count | Integer | The number of messages delivered to the FeedMaster. Another field named JMSXDeliveryCount reports the same statistic. |
Monitoring API Usage for Tasks
You can use the following APIs to monitor historical usage of concurrent and daily APIs (Triggered and Ultra Tasks) by running the Pipeline runtime API with the api-stats argument.
Permissions
Public APIs for collecting API usage statistics require Org admin access.
Viewing Concurrent and Daily API Metrics
Endpoint
GET /api/1/rest/public/runtime/api-stats/<Org>
Description
Returns concurrent and daily API stats for your Tasks in the specified Org.
Parameters
Parameter Name | Data Type | Description | Default Value | Example |
---|---|---|---|---|
Org | String | Required. The name of the Org. | N/A | DevOrg |
Date_time | String | The start timestamp in milliseconds. | N/A | 2020-06-20T01:01:01 |
Request Example
GET /api/1/rest/public/runtime/api-stats/DevOrg?date_time=2020-06-24T19:39:58
Response Example
{ "response_map": { "daily_limit": 10000, "organization": "DevOrg", "concurrent_call_in_progress": 3, "concurrent_limit": 10, "daily_call_used": 6 }, "http_status_code": 200 }
Viewing Daily API Metrics
Endpoint
GET /api/1/rest/public/runtime/api-stats/<Org>/daily
Description
Returns daily API metrics for your Tasks in the specified Org.
Parameters
Parameter Name | Data Type | Description | Default Value | Example |
---|---|---|---|---|
Org | String | Required. The name of the Org. | N/A | DevOrg |
Date_time | String | The specific time period in UTC. Format: %Y-%m-%dT%H:%M:%S. | N/A | 2020-06-20T01:01:01 |
Request Example
GET /api/1/rest/public/runtime/api-stats/DevOrg?date_time=2020-06-24
Response Example
{ "response_map": { "top_tasks": [ { "task_owner": "jdoe@beignet.com", "task_id": "5x14p1b6d171a0015d6049d", "task_name": "Convection1", "task_path": "/DevOrg1/jd/shared", "api_count": 5 }, { "task_owner": "autostart21W@gmail.com", "task_id": "p3b5c5ts23b21cmeen44aa59", "task_name": "Convection2", "task_path": "/DevOrg180/projects/MyAPI", "api_count": 3 } ], "task_count": 2, "daily_call_used": 6, "date": "2020-04-24", "organization": "DevOrg180", "daily_call_available": 9994 }, "http_status_code": 200 }
Viewing Concurrent API Metrics
Endpoint
GET /api/1/rest/public/runtime/api-stats/<Org>/concurrent
Description
Returns concurrent API metrics for the Tasks in a specified Org.
Parameters
Parameter Name | Data Type | Description | Default Value | Example |
---|---|---|---|---|
Org | String | Required. The name of the Org. | N/A | DevOrg |
Date_time | String | The specific time period in UTC. Format: %Y-%m-%dT%H:%M:%S. | N/A | 2020-06-20T01:01:01 |
Request Example
GET /api/1/rest/public/runtime/api-stats/DevOrg/concurrent?date_time=2020-06-24T19:39:58
Response Example
{ "response_map": { "concurrent_call_in_progress": 3, "organization": "DevOrg", "tasks": [ { "task_path": "/DevOrg/jd/shared", "task_owner": "jdoe@beignet.com", "task_name": "Concurrent1", "task_id": "5ed14b1b9d173a0015d4089d", "api_count": 3 } ], "time_stamp": "2020-06-24T19:39:54" }, "http_status_code": 200 }
Task Monitoring Field Descriptions
Use the following table to understand the Field Keys in the returned response to the API.
Field Key | Type | Description |
---|---|---|
Org | String | The name of the Org. |
Date | String | Get concurrent and daily stats at specific time in UTC in the following date time format:%Y-%m-%dT%H:%M:%S |
Daily limit | Integer | The limit of APIs for the day. |
Daily Call Used | Integer | The number of total API calls |
Concurrent Limit | Integer | The limit of concurrent usage for today |
Concurrent Calls in Progress | Integer | The concurrent API calls in progress. |
Task Count | Integer | The number of Tasks that consumed API Calls on the given date. |
Top Tasks | List | The top 10 APIs that consumed API calls. |
Task name | String | The name of the Task. |
Project | String | The Project path to the Task. |
Owner | String | The owner of the Task. |
API Call Count | Integer | The total number of API calls made. |
API Calls for Controlling Pipeline Execution
Manually Stop a Pipeline
POST /api/1/rest/public/runtime/stop/<Org>/<ruuid>
Stops the running Pipeline with the specified Runtime ID.
Manually Resume a Resumable Pipeline
POST /api/1/rest/public/runtime/start/<Org>/<ruuid>
Resumes a Suspended Resumable Pipeline with the specified Runtime ID.
Pipeline and Task Monitoring API Examples
Get the Runtime Information of Pipelines and Tasks
The following queries return information about the executed Pipelines:
Scenario | Query |
---|---|
Return the status of executed Pipelines for the last hour within the Org. | GET https://elastic.snaplogic.com/api/1/rest/public/runtime/orgname |
Return the status of executed Pipelines for the specified time within the Org. | GET https://elastic.snaplogic.com/api/1/rest/public/runtime/orgname?start=&end=> |
Return the status of executed Pipelines for last specified number of hours. | GET https://elastic.snaplogic.com/api/1/rest/public/runtime/orgname?last_hours=4 |
Return the concurrent and daily metrics for API usage for today. | GET /api/1/rest/public/runtime/api-stats/prod_org |
Get Pipeline Information
The following queries return information about the specified Pipeline runtime:
Scenario | Query |
---|---|
Return the summary information of the pipeline with the specified runtime ID. | GET https://elastic.snaplogic.com/api/1/rest/public/runtime/orgname?level=summary |
Using REST Get Snap
You can also access these APIs using the REST Get Snap with a Basic Auth account configured to access SnapLogic. The following image shows a query for only those Pipelines that are in the Queued state.
See Also
- REST GET and the SnapLogic Public APIs for Pipeline Executions
- How to extract full runtime information from pipelines
- How to obtain error reasons, resolutions, and so on, using the runtime ID of a pipeline
- How to integrate third-party job schedulers to kick-off and monitor SnapLogic pipeline execution
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.