Highlights

The SnapLogic August 2024 Release includes the following highlights:

Admin Manager

• Enable or disable the SnapLogic Cloud Pattern Library

API Management

• New Public API to invalidate response cache for API proxies

• Support for Secrets Management tools

Documentation Enhancements

• New guide for organizations planning to migrate their environments from the US to the EMEA control plane

• A new Introduction section replaces the dated Getting Started guide

GenAI App Builder

• Amazon Bedrock LLM: Added the Agent Retrieve, Agent Retrieve and Generate, and Invoke Agent Snaps

• Azure OpenAI LLM: Added the Azure OpenAI Entra ID Application Account, Azure OpenAI Entra ID User Account, and Azure OpenAI Entra Token Accounts

• Google GenAI LLM Snap Pack

• OpenSearch: Added the Amazon OpenSearch Account

• OpenAI LLM: Added the Add Message, Create and Run Thread, Create Thread, and Run Thread Snaps.

Monitor

• New design for the System overview

Platform

• Additional rules added to the public API to check the pipeline quality

Snaps

• Databricks - Run Job

• Google Directory - Google Service Account

• Oracle - Parallel Load

Dot Releases – Snaps

Upgrading to this Release

Admin Manager

New Features

• Enable or disable the SnapLogic Cloud Pattern Library for the environment. Learn more.

  Open

  

API Management

New Features

• New Public API to invalidate the response cache defined in the HTTP Response Cache policy. Learn more.

Enhancements

• Support for Secrets Management: If you use secrets management tools and services to manage your API Keys, you can now reference them when configuring API policies. The following policies have expression-enabled fields for API Keys:

  • API Key Authenticator

  • OAuth Policy

• API logs: The logs now contain more details about failed requests. When investigating why a request failed, you can now identify which Snaplex node ran the policy that failed the request.

Fixed Issues

• Fixed an issue where the Request Transformer policy did not get applied during the API call.

• Fixed an issue where headers were not actually modified by the API Proxy’s mapping rules.

• Fixed an issue where the Authorization Header in the outbound account would add an extra space between the scheme and the bearer token making some upstream servers recognize the token as invalid, including Task authentication.

Known Issues

• When you create an API from an existing Project and choose the Assets and Snaplex in the dialog, API creation is successful, and all the assets import successfully. However, Triggered and Ultra Tasks fail to recognize the designated Snaplex that you select during API creation in the UI. Even if you explicitly specify the plex_path in the public API, the tasks will default to using the Snaplex associated with the source project.
  Workaround: Set the Snaplex for each task manually.

• In APIM, the Create a Version and Create an API dialogs do not display the Create from Git Checkout checkbox, even if you are authenticated in Git.

AutoSync

Fixed Issues

• When you save a new merged column without changing the name and then change it and save again, the changed name is now preserved.

• When you apply both concatenation and data masking transformations, the data masking is applied correctly.

Known Issues

• After you resolve a naming conflict, the transformations data preview in either the wizard or the editor might show the wrong table names or duplicate table names.

• The data preview for the Replace empty values screen displays data from the wrong table when the selected table has no nullable columns.

Data Automation

Enhancements

• Enhanced the ELT Execute Snap to display statistics for all SQL statements executed. The statistics are shown in both the pipeline execution statistics and the output view of the Snap. For target Databricks Lakehouse Platform (DLP) tables, the Snap also allows you to download DML statement execution statistics in a JSON file (extraStats).

• Enhanced the pipeline execution statistics of the ELT Insert-Select Snap to be displayed in its output view. For target Databricks Lakehouse Platform (DLP) tables, you can download a JSON file that includes additional statistics (extraStats) on DML statement executions.

• Enhanced the ELT Load Snap to support loading data from nested AVRO, JSONLines, ORC, or Parquet files in Azure storage to target tables in a Databricks Lakehouse Platform (DLP) instance. Only two Load actions are supported: Drop and create table and Append table.

• Upgraded the jOOQ library for the ELT Snap Pack from v3.9.1 to v3.17.x.

Fixed Issues

• Some conflicts between internally used Azure libraries prevented the ELT Load Snap from reading files using an ADLS Gen 2 account connection.

• The ELT Merge Into Snap’s SELECT SQL statement could not fetch the target tables' information from the PG_TABLE_DEF catalog table of the Amazon Redshift instance.

  • The SELECT SQL statement now uses lowercase for both non-delimited and delimited table names and excludes double quotes from delimited table names to fetch the values from the catalog table.

  • PG_TABLE_DEF is a Redshift system catalog table that contains information about the tables such as table names, column names, data types, and other metadata.

Documentation Enhancements

API Management

• Updated the Generic OAuth2 Policy with the OAuth2 framework for the authorization code grant type. Learn more.

New Content

• Use case article with pipeline samples about Gen AI App Builder pipelines that read and parse content from a PDF source and deliver responses to prompts about the information.

• Configuration of Managed identities for the Azure Storage Blob service in the Azure portal.

• A new Topics by task page provides quick links to concepts and procedures for common tasks.

• The new Introduction section of the documentation replaces legacy Getting Started content. It contains:

  • An overview of the SnapLogic Platform and pipeline execution.

  • An overview of the workflow for SnapLogic apps, Designer, AutoSync, Monitor, Admin Manager, SnapGPT, and GenAI App Builder.

  • The tutorial: Create your first pipeline.

  • Information about configuring user settings and multi-factor authentication (MFA).

New Videos

• Learn how to set up Azure Managed Identity in the Azure Portal:

  Open

  

• Learn how to Configure Azure Storage Account with a Managed Identity in SnapLogic:

Content Published on New Doc Site

The following content was added to the new site for this release. Visit https://docs.snaplogic.com to view all content available on the new site.

• Classic Manager Project Assets

• Groundplex Node Management

• Snaplex Monitoring

• Ultra Task Pipelines

GenAI App Builder

New Features

• Amazon Bedrock LLM: Added the following Snaps to the Snap Pack:

  • Agent Retrieve: Performs a query on a Knowledge Base and retrieves information from it based on the specified prompt.

  • Agent Retrieve and Generate: Performs a query on a specific Knowledge Base for relevant context and uses a designated model to generate responses based on the retrieved information.

  • Invoke Agent: Runs the Amazon Bedrock Invoke Agent operation on the Amazon Bedrock service.

• Azure OpenAI LLM: Added the following accounts to the Snap Pack:

  • Azure OpenAI Entra ID Application Account: Uses applications to access Azure resources securely with application-level permissions for service principal authentication.

  • Azure OpenAI Entra ID User Account: Authorizes with the user/delegate level permissions associated with the logged-in user profile.

  • Azure OpenAI Entra Token Account: Uses token-based authentication to interact with Azure OpenAI APIs.

• Google GenAI LLM: Introduced this Snap Pack, which contains the following Snaps and Accounts:

  • Google Gemini Embedder: Generates an embedding vector based on the provided input data.

  • Google Gemini Generate: Generates text for tasks such as content creation, response generation, and more.

  • Google GenAI Prompt Generator: Generates the augmented user prompt from the specified prompt template using the mustache template format.

  • Google Gemini API Key Account: Authenticates with Google AI's APIs using the specified API key for secure and authorized access.

  • Google GenAI Access Token Account: Authenticates with Google AI's services using an access token.

  • Google GenAI Service Account: Authenticates with Google AI's services using a service account for programmatic access.

• OpenAI LLM: Added the following Snaps to the Snap Pack:

  • Add Message: Adds a message to the specified thread using the Assistant API.

  • Create and Run Thread: Creates and runs a thread after assigning the assistant to respond to the newly created thread.

  • Create Thread: Creates a thread using the Assistant API.

  • Run Thread: Assigns the assistant to provide a response to the existing thread.

• OpenSearch: Added the following Account to the Snap Pack:

  • Amazon OpenSearch Account: Authenticates with OpenSearch services using the specified Access key ID and Secret access key. It also supports the Cross-account IAM role.

Enhancements

Amazon Bedrock LLM:

• Enhanced the Amazon Bedrock Prompt Generator Snap to output the entire prompt message array and user role selection for the prompt.

• Enhanced the Anthropic Claude on AWS Messages Snap with a message payload to generate responses based on a specified list of messages.

Azure OpenAI LLM:

• Enhanced the Azure OpenAI Prompt Generator Snap to output the entire prompt message array and user role selection for the prompt.

• Enhanced the Azure OpenAI Chat Completions Snap with a message payload to generate responses based on a specified list of messages.

OpenAI LLM:

• Enhanced the OpenAI Prompt Generator Snap to output the entire prompt message array and user role selection for the prompt.

• Enhanced the OpenAI Chat Completions Snap with message payload to generate responses based on a specified list of messages.

Monitor

Enhancements and Changes

• The new design for the System overview graph node view shows you all that you need to know about the environment infrastructure at a glance:

   

  • The bars represent the average memory, CPU utilization, and disk utilization for each node for the selected time period.

  • The colored carets mark the maximum values during the selected time period.

  • Bars and carets are color-coded for usage: Green indicates up to 45%, yellow indicates up to 75%, and red indicates over 75%.

  • Snaplexes with any nodes where the average or maximum values are either yellow or red show at the top of the page.

• In the graph node view, hover over a node to view the actual values:

• Now, when you switch from the System overview node graph view to the list view, the Snaplex list opens instead of the Node list.

• In the Execution overview, the search Invocation type filter now provides options such as nested, error, and regression pipelines. The following screenshot shows the complete list:

• The node details panel's Additional details tab no longer includes Max Threads because it doesn’t apply to the Windows OS and was inaccurate for the Linux OS.

Learn more about:

• The System overview graph view

• The System overview list view

• The Snaplex details panel

• The node details panel

Platform

New Features

• The public API for pipeline quality includes additional rules to provide complete reports for all standard checks. The reports include message levels with actionable message descriptions. Learn more.

• A new guide for organizations planning to migrate their environments from the U.S. to the EMEA control plane.

Enhancements

• A new note in the UI for the Update Snaplex dialog box in the Downloads tab states that only the latest version download links are available because deprecated versions cannot be used for installation. Learn more.

• The Automated Pipeline Killer prevents high memory-consuming pipelines from causing node crashes. It tracks pipeline use and reservation of heap and non-heap memory. When the committed and used memory is excessive, the Automated Pipeline Killer identifies the faulty Snap responsible for the high memory utilization, kills that pipeline, and adds an error message to the Snap. The enhanced Automated Pipeline Killer:

  • Considers non-heap memory versus heap memory and committed memory versus used memory.

  • Adds information to the node log about the utilization of heap and non-heap memory, memory pools, and buffer pools.

  • Improves memory tracking by considering the ratio of total committed memory to physical memory to prevent an unresponsive JCC state.

  • Improves the pipeline termination process by ignoring completed Snaps and considering only running Snaps. The Pipeline Execution Statistics indicate details in the UI.

The Automated Pipeline Killer is enabled for all environments by default and has a threshold setting that controls its behavior. To adjust the threshold, contact your CSM. Learn more.

Fixed Issues and Behavior Changes

• Fixed an issue with the OAuth2 password grant type to make sure that the Snap OAuth2 Account token refreshes successfully every 20 minutes.

• Updates to Accounts using OAuth2 now succeed without issues.

• For Accounts using OAuth2, the Platform no longer saves data in the refresh token field unless the endpoint returns refresh tokens.

Known Issues

• Snaplexes configured with the French or Macedonian locale may not be able to make some HTTP requests to the control plane.
  Workaround: Roll back the impacted Snaplex to main-24152 4.37 GA (Deprecated) until we fix the issue.

• In Snaplex version 4.38, pipelines using the Catalog Insert Snap no longer populate the Data Catalog tables in Manager with metadata.
  Workaround: Do not use Snaplex version 4.38 to run pipelines with the Catalog Insert Snap.

• When you call a Triggered Task on a Groundplex using the Cloud URL with a payload of 26 MB or larger, the pipeline processing can exceed the WebSocket connection timeout.
  While the pipeline succeeds in delivering the payload, the bytes after the 26MB limit might be lost on the response side.
  If the payload is on the request side, then only the first 26 MB of bytes are sent to the pipeline, causing the possibility of pipeline failure.

Snaps

Upcoming

New Snaps/Accounts

• Databricks - Run Job: Automates the execution of tasks or processes in the Databricks workspace.

• Google Directory - Google Service Account: Uses a Google Service Account to connect Google Directory Snaps to a data source.

• Oracle - Parallel Load: Executes a high-performance data load by inserting data in parallel across multi-threaded tasks.

Enhancements

• Apache Velocity is upgraded from v1.7 to v2.3 across all Snap Packs to fix the vulnerability issue in v1.7.

• API Suite:

  • Updated the API Suite Snaps to process authentication information from incoming documents to support features such as credentials stored in a secrets manager.

• Binary:

  • Enhanced the Azure Storage Account with Managed Identity that enables you to integrate with Azure Blob service to access Azure resources. Learn more about Configuration of Managed Identities in Azure Portal.

• Flow:

  • Enhanced the Pipeline Execute Snap to evaluate the Pipeline Parameters for child pipelines one time at the beginning of the batch processing or during instance reuse, thereby improving performance.

• MongoDB:

  • Enhanced the MongoDB - Atlas Vector Search Snap with the following:

    • Suggestions for the Search index field that enables the Snap to populate the associated indexes in the list.

    • The input schema displays the mandatory vector field and optional filter suggestion (if the Search index contains a filter type query) in alignment with the fields expected by the Snap.

• ML Data Preparation:

  • Enhanced the Date Time Extractor Snap to support Date time formats (YYYY-MM-dd HH:mm:ss and YYYY-MM-dd HH:mm:ss.SSS) and allow the root path to auto-convert all fields.

• Oracle:

  • Enhanced the Oracle Insert, Merge, Update, and Delete Snaps to support the Enable, Disable, and Force commands in the Session parameters that enable you to dynamically manage specific session settings.

• PostgreSQL:

  • Enhanced the Postgres Account and Postgres Dynamic Account with support for IAM and Cross-account IAM role authentication.

• Redshift:

  • Upgraded the JDBC driver for the Redshift Snap Pack to v2.1.0.29 to address the SQL Injection vulnerabilities. Pipelines using the Redshift Snaps are not impacted after the driver upgrade, because the latest JDBC driver is fully backward compatible.

• Snowflake:

  • Enhanced the Snowflake—Select Snap with Time Travel query support that enables you to access historical data at any point in a defined time period. This includes restoring data-related objects, duplicating and backing up data, and analyzing data usage.

• Upgraded the org.json.json library from v20090211 to v20240303 for the following Snap Packs, which is fully backward-compatible: 

Fixed Issues

• When the Router Snap was the first Snap in a triggered task, the documents were not processed. Previously, the response contained a 200 status code but was empty. Now, it processes one document and provides non-empty output.

Behavior Changes

• Previously, the Router Snap generated no output without an input document. Now, the Snap generates an empty document when no input document is provided.

• Fixed an issue in the Snap OAuth2 Accounts for endpoints that do not support auto-refresh. Previously, you could enable Auto-refresh even though it would fail to refresh the access token. Now, on account validation, an alert advises that you cannot auto-refresh because the endpoint does not support it. Existing Accounts with the auto-refresh setting enabled continue to work as before.

Known Issues

• SAP Snap Pack: When you upgrade the SAP Snap Pack from 4.37 main24152 to 4.38 main27765 (from SnapLogic Manager), with the old SAP native library still installed, the following fatal initialization error displays:
  Pipeline prepare error Reason: Dependent native libraries are not found or could not be resolved: Could not initialize account. [com-snaplogic-snaps-sap-jcoaccount_2-438rc27664]: Native library sapjco3 is too old. Found library /opt/snaplogic/ldlib/libsapjco3.so has version "753.811", but required is at least version "753.1324".

  Workaround:
  Follow these steps to resolve the issue:

1. Copy the updated native library on every node in the Snaplex:

   • For Linux, copy the libsapjco3.so file to the /opt/snaplogic/ldlib folder.

   • For Windows, copy the sapjco3.dllfile to the C:\opt\Snaplogic\ldlib folder.
     Note: For Snaplexes with multiple nodes, make sure to install the library on every node.

2. Restart all the nodes in the Snaplex.

You can download the library here.

• Hadoop Snap Pack: The upgrade of the Azure Storage library from v3.0.0 to v8.3.0 caused the following issue when using the WASB protocol:
  When you use invalid credentials for the WASB protocol in (HDFS Reader, HDFS Writer, ORC Reader, Parquet Reader, Parquet Writer) Snaps, the pipeline does not fail immediately, instead it takes 13-14 minutes to display this error: reason=The request failed with error code null and HTTP code 0. , status_code=error
  SnapLogic is actively working with Microsoft Support to resolve the issue.

SnapGPT

Enhancements

• You can now resize the SnapGPT pane to make it easier to read the responses to your prompts.