Groundplex Deployment for Kubernetes and Elastic Scaling Support

In this article

Overview

You can deploy Snaplex instances in a Kubernetes environment. In this deployment, you host the resources, and the Snaplex type is a Groundplex, which you also administer, taking advantage of the orchestration that Kubernetes provides. This support extends to the Kubernetes Horizontal Pod Autoscaler (HPA) resources. 

Starting in the 4.25 Release (May 2021), the new feature enables you to set up your Groundplex nodes in your Kubernetes environment with a Helm Chart that defines the node configuration for discoverability and the dynamic allocation of CPU resources, memory, and disk space. 

This article explains how you can deploy and configure SnapLogic Snaplex nodes in a Kubernetes environment and contains an attached Helm Chart zip file you can use.

This solution is distinct from the previous one supported in prior releases—the Helm Charts, methodology, and deployment steps are completely different.

Even though the previous solution is still supported, we recommend that you deploy your Groundplex nodes in Kubernetes using the Helm Charts in this article. To take advantage of Elastic Scaling on existing Groundplex nodes, you must redeploy them using the Helm Charts contained in this article.


Key Features

  • Graceful termination: Resources for the JCC nodes in the Groundplex now gracefully terminate when Pipeline executions complete.

  • HPA support in Helm Chart: You can use the Horizontal Pod Autoscale algorithm to define custom performance metrics.

  • CPU/Memory utilization metrics: You can now view the average of these metrics observed on Snaplex nodes can be collected through the Metrics Server.

  • Dashboard Support: You can set up the Grafana application monitoring for the scaling up and down of resources.

Summary of Implementation

Elastic Scaling is based on the Kubernetes Horizontal Pod Autoscaling, which operates on the ratio between the desired metric value and current metric value. The implementation requires the K8 Metrics Server and Prometheus adapters, both open source software common in Kubernetes. Grafana is an optional component that provides the dashboard monitoring for the scale up and scale down activities.

Workflow

  1. Org admin configures Groundplex nodes.
  2. Kubernetes Admin deploys Helm Chart.
  3. Groundplex node resources spin up and down depending on need.
  4. Org admin monitors Pipelines and Groundplex node resources through SnapLogic Dashboard.
  5. Kubernetes Admin monitors Elastic Scaling through analytics dashboard like Grafana.

Prerequisites

  • You must install the following software:
    • Metrics Server from GitHub 
    • Prometheus  (Recommend Helm Chart version:  14.0.1)
    • Prometheus Adapter (Recommend Helm Chart version: 2.12.1)
    • Grafana
  • You should have familiarity with the following topics:
    • Provisioning Snaplex Nodes in Kubernetes using Helm Charts
    • Horizontal Autoscaling
    • Prometheus

Support and Limitations

The customized metric for queued Pipelines may not autoscale properly when Snaplex-based Scheduler is enabled for the Org.

Known Issues

None