Third-party Observability Tools Integration
In this article
Overview
IT organizations often leverage centralized tools to monitor their systems. Typically, these centralized tools enable IT groups to do the following:
Avoid reliance on the tool to monitor itself.
Observe reduced Total Cost of Ownership (TCO) to manage the monitoring, alerts, and notification system in one place.
Manage the history by the retention of the data for audits.
The SnapLogic® Third-party Observability (Public Preview) feature offers the capability to integrate pipeline runtime logs with your third-party monitoring tools. This feature enables your IT organization to track, troubleshoot, analyze, and optimize your integrations in production environments.
The SnapLogic platform uses OpenTelemetry (OTEL) to support telemetry data integration with third-party observability tools. This public preview feature release includes the implementation of the service to enable you to monitor your pipeline execution runtime logs in Datadog and New Relic.
This feature requires a subscription. Contact your CSM to turn on the Open Telemetry feature.
Certified Third-Party Observability Tools
Datadog
New Relic
This integration solution is designed to be compatible with any vendor or open-source Observability tool that supports the OpenTelemetry Collector-based approach for the collection, processing, and export of telemetry data. Due to the extensive variety of tools available in the observability space, we currently do not plan to certify every tool on the market.
Prerequisites
Groundplexes are installed and set up.
The Open Telemetry feature subscription must be enabled for your Org/Environment
Support for Cloudplexes is on the product roadmap.
Implement OTEL Services
OpenTelemetry is an open-source, vendor-agnostic observability framework and toolkit designed to create and manage OpenTelemetry data. For the SnapLogic application, you can capture metrics and logs. OTEL reporting sends messages of Error severity levels to your connected 3rd-party monitoring tools. Messages of all severity levels are retained in the logs.
How OpenTelemetry Works
The OpenTelemetry Collector receives, processes, and exports OpenTelemetry data. When you deploy the OpenTelemetry service, the Collector retrieves the data from the JCC node logs and routes them to your third-party monitoring tool using the OpenTelemetry Protocol (OTLP).
The following three components comprise the OpenTelemetry Collector:
Receiver - Defines how the logs are sent to the OpenTelemetry Collector.
Processor - Defines the method of log retrieval. In most cases, batch mode is preferred.
Exporter - Specifies the third-party tool.
The diagram below provides shows the architecture of the OpenTelemetry service that is implemented in the SnapLogic Platform.
This document describes only the retrieval of logs sent in and supported by the OpenTelemetry format, not the original log format from the Groundplex JCC node. Support for capturing the log data directly from the Groundplex JCC node is on the product roadmap.
Workflow
Set up the OpenTelemetry Service.
Run your pipelines.
Observe pipeline runtime data in the monitoring tool.
Datadog Monitoring Tools Support Workflow
This page describes how to perform those steps to monitor Groundplexes deployed on Windows, Linux, or Docker. Refer to Deploy OTEL as a Daemonset for Groundplexes running on Kubernetes.
Install the OpenTelemetry Package
Download the OpenTelemetry Collector Contrib package.
Save the package on the same JCC node of the machine that hosts the Groundplex.
Configure the OpenTelemetry Services
Create the YAML configuration file.
You can use the following template:
Specify the gRPC URL in the YAML template. This is in the environment variable in your host machine.
For batch-mode processing of the logs, define the following values for the processors.
send_batch_max_size
: 100send_batch_size
: 10timeout
: 10s
For
exporters
, add the values for your Datadog API:site
: "http://datadoghq.com "key
: ${env:DD_API_KEY}
The following is an example of a YAML configuration file: