Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Open an existing Snaplex in the Org:

    1. Navigate to the target Snaplex in Manager.

    2. Click on the Snaplex name to display the Update Snaplex dialog.
      Alternatively, if none exists, Create a Snaplex

  2. On the Downloads tab, download the Configuration file using the option image-20240624-095938.png or use the API api/1/rest/public/snaplex/config/<path_to_groundplex> to download the slpropz file.

    update-snaplex.png

  3. Click Cancel to exit the dialog.

  4. To ensure that the Kubernetes pods continue to run:

    Use the API to retrieve the information needed to configure a node on a Groundplex.
    GET https://{controlplane_path}/api/1/rest/public/snaplex/config/{plex_path}?{query_parameters}.
    An example would be similar to the path provided below: https://elastic.snaplogic.com/api/1/rest/public/snaplex/config/snaplogic/shared/MyGround

    1. Use the path retrieved from the API in the snaplogic_config_link parameter of the Helm chart. Learn more at Retrieve config info for a Groundplex. and Download the slpropz configuration file

  5. Set the parameter snaplogic_secret in the Helm chart YAML file to the name of the Kubernetes secret you create, as described in the Deploy the Helm Chart section.

Info

For a Zero Trust Kubernetes Installation, open up port TCP 443 and websockets connections for the sites mentioned below:
https://elastic.snaplogic.com
https://tcp.elastic.snaplogic.com
https://snaplogic-prod-sldb.s3.amazonaws.com
https://s3.amazonaws.com

...

You can associate Org admin credentials with the SnapLogic secret created when enabling enhanced encryption. This makes it easier to share the Snaplex service with your organization's users. The SnapLogic Org admin credentials and the enhanced encryption secret are in JSON format as key/value pairs.

  1. Generate a key and encode it for each value.
    NOTE: We recommend that you use Base64 to encode the values.

  2. To create the SnapLogic secret:

    1. Create the YAML file with the following two keys: username and password.

      Example YAML File

      Code Block
      apiVersion: v1
      kind: Secret
      metadata:
        name: mysecret
      type: Opaque
      data:
        username: <Base64 username>
        password: <Base64 password>

      TIP: Run the following command to encode your username/password into the text of the secret:
      $ echo -n "snaplogic_username_or_password" | Base64

      IMPORTANT: If your password includes any of the following characters, you must escape the character with a backslash (\) in the string that you pass to the encoder:
      \ (backslash)
      $ (dollar sign)
      ' (apostrophe or single-quote)
      ` (backtick)
      " (double-quotes)
      & (ampersand)
      | (pipe symbol)
      ; (semicolon)
      ! (exclamation mark)
      For example, if your password is mypa$$word, pass the string mypa\$\$word to the Base64 encoder.

    2. Run the following command:

      $ kubectl apply -f snaplogic_secret.yaml

     

  3. (Optional) If Enhanced Encryption is enabled for your Org, create the Enhanced Encryption secret by running the following commands:
    $ kubectl create secret generic enhanced-encryption-secret --from-file=keystore_jks --from-file=keystore_pass
    $ kubectl apply -f enhanced_encryption_secret.yaml

  4. After the secret is created, delete the YAML file because it is no longer needed.

...

The following list describes each field parameter in the values.yaml file:

parameters

Field

Description

Guidelines

Regular nodes count

Specifies the number of JCC deployed to run the SnapLogic container.

If HPA is enabled, then the minReplicas and maxReplicas counts are used from values.yaml

FeedMaster nodes count

Specifies the Specifies the number of FeedMaster Pods to deploy.

 N/A

Docker image of SnapLogic Snaplex image

Specifies the repository where the image resides and the tag indicating its the version of the image. Although you can specify a version of your Snaplex, we recommend entering the latest version for that you enter the latest version for the most recently released SnapLogic build.

 N/A

SnapLogic configuration link

Specifies the link to the SnapLogic JCC configuration file (also known as .slpropz).

 N/A

SnapLogic Org admin credential

Specifies the secret (an encoded username and password) to authenticate the deployment.

 N/A

Enhanced encryption secret

Specifies the secret (an additional encoded username and password) to authenticate the deployment, available only to the user.

 N/A

CPU and memory limits for the nodes

Specifies the upper limits and requests for the CPU and memory resources. You can set these values for only the upper limits. The lower limits are system-defined and cannot be modified.

Limits: The maximum amount of resources (CPU and memory) that a container can consume.

Requests: The minimum amount of resources that a container needs to function properly.

This ensures that the container is always scheduled on a node that can fulfill its minimum requirements.

Default file ulimit and process ulimit

Specifies the number of possible open file descriptors and processes for the JCC containers.

The value should be greater than the number of slots configured for the node (Maximum Slots under the Snaplex's node properties).

If not set, then the node defaults will be used.
(/etc/security/limits.conf).

Probes

Monitors the SnapLogic application.

 

Autoscaling

Sets autoscaling properties.

By default, enabled to False.

 

Enabled to false by default. Contact SnapLogic CSM for recommendations for this setting.

The HorizontalPodAutoScaler is enabled by setting this property to true.
Contact SnapLogic CSM for recommendations for this setting.

minReplicas

maxReplicas

minReplicas defines the minimum number of Pods that must be running.

maxReplicas defines the maximum number of Pods that can be scheduled on the node(s)

The general guideline is to start with 1:2 or 1:3 Pods per node.

targetAvgCPUUtilization

This is the average CPU utilization across all Pods. HPA will scale up or scale down Pods to maintain this average.

 N/A

targetAvgMemoryUtilization

This parameter specifies the average memory utilization (as a percentage of the requested memory) that the HPA should maintain across all the replicas of a particular deployment or stateful set.

 N/A

scaleDownStabilizationWindowSeconds

This parameter controls the amount of time the HPA waits (like a cool-down period) before scaling down the number of pods after a decrease in decreasein resource utilization.

 N/A

Termination Grace Period Seconds

The time to shut down a Pod node gracefully before it gets terminated.This is an upper limit for the time to gracefully shut down a Pod.

N/A

IPv6

Enabled to false False by default. You can enable IPv6 for your connections if your infrastructure supports it.

 N/A

Info

The regular nodes count and feed master nodes count are similar to the minReplicas and maxReplicas fields, which refer to the number of Pods needed to run the application. These are used to configure the pod count statically rather than using autoscaling, as in the case of minReplicas and maxReplicas.

The YAML files set the default name values for Groundplex nodes. The example provided in this document snaplogic-snaplex is only for reference and not a required naming convention.

Example Helm Chart

Code Block
# Default values for snaplogic-snaplex.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Regular nodes count
jccCount: 1
# Feedmaster nodes count
feedmasterCount: 1
# Docker image of SnapLogic snaplex
image:
  repository: snaplogic/snaplex
  tag: latest
# SnapLogic configuration link
snaplogic_config_link:
# SnapLogic Org admin credential
#snaplogic_secret:
# Enhanced encryption secret
#enhanced_encrypt_secret:
# CPU and memory limits/requests for the nodes
limits:
  memory: 8Gi
  cpu: 2000m
requests:
  memory: 8Gi
  cpu: 2000m
# Default file ulimit and process ulimit
sl_file_ulimit: 8192
sl_process_ulimit: 4096
# Enable/disable startup, liveness and readiness probes
probes:
  enabled: true
# JCC HPA
autoscaling:
  enabled: false
  minReplicas:
  maxReplicas:
  # Average count of Snaplex queued pipelines (e.g. targetPlexQueueSize: 5), leave empty to disable
  # To enable this metric, Prometheus and Prometheus-Adapter are required to install.
  targetPlexQueueSize:
  # Average CPU utilization (e.g. targetAvgCPUUtilization: 50 means 50%), leave empty to disable.
  # To enable this metric, Kubernetes Metrics Server is required to install.
  targetAvgCPUUtilization:
  # Average memory utilization (e.g. targetAvgMemoryUtilization: 50 means 50%), leave empty to disable.
  # To enable this metric, Kubernetes Metrics Server is required to install.
  targetAvgMemoryUtilization:
  # window to consider waiting while scaling up. default is 0s if empty.
  scaleUpStabilizationWindowSeconds:
  # window to consider waiting while scaling down. default is 300s if empty.
  scaleDownStabilizationWindowSeconds:
# grace period seconds after JCC termination signal before force shutdown, default is 30s if empty.
terminationGracePeriodSeconds: 900
# Enable IPv6 service for DNS routing to pods
enableIPv6: false

...

After your Snaplex and FeedMaster nodes are deployed, you can start designing and running Pipelines and Tasks.

Installing utilities

For Kubernetes-based deployments, users have to build images/containers that install the utilities in appropriate locations. Those images can use the official Snaplogic image as the base image. When deploying the Snaplex to Kubernetes, the users would then use that image/container for deployment and have necessary dependencies and utilities in place.

Best Practices

  • Avoid running processes in the same container as the JCC so that the JCC has the maximum amount of memory available, as requested.

  • Do not overwrite the global.properties file unless you are working with your CSM to customize your Groundplex.

  • Request resources upfront. The requests determine the minimum required resources, while limits set the maximum resources a Container can consume. Setting them to the same amount ensures stability and exact resource usage. To do this, set the pod’s request and limit to the same value, as shown in the image below:

...