Highlights

The SnapLogic May 2024 release includes the following:

Admin Manager

• Collapsible navigation pane

API Management

• New public API for Git Integration Operations for API version

• API Metrics now available in Monitor

• New public API to perform soft delete on the API and API version

• A JWT Outbound policy is now available for Proxies

AutoSync

• Merge columns transformation to concatenate data

• Interface to update data pipelines to the latest AutoSync Snap Pack

• New target endpoints: Databricks, Amazon S3, and Azure Data Lake Storage (ADLS) Gen2

Data Automation

• Load flat and nested data sets from Parquet files to Snowflake with the ELT Load Snap

Documentation Updates

• ML Analytics and ML Natural Language Processing Snap Packs are published on the new Documentation site.

• Video on adding a Response Transformer Policy to a Proxy.

GenAI Builder

• Added the Anthropic Claude on AWS Messages Snap to the Amazon Bedrock Snap Pack

Monitor

• New pages:

  • Alert history

  • Integration catalog

Platform

• New Git Integration public API to discard local changes

• New public API to create, update, and delete a Groundplex

• Modified public API for sending a notification email when a user is added

Snaps

• The Azure SQL Bulk Extract Snap now supports the Azure SQL Entra and Azure SQL Entra Dynamic Accounts

• Added support for Geospatial data types (Geography and Geometry) for the SQL Server Snap Pack

• Added the MongoDB Execute and MongoDB - Atlas Vector Search Snaps to the MongoDB Snap Pack

• Added the NetSuite Add List and NetSuite Async Add List Snaps to the NetSuite SOAP Snap Pack

• Added Parquet Parser and Parquet Formatter Snaps to the Transform Snap Pack

SnapGPT

• Improved UX

Dot Releases - Snaps

Upgrading to this Release

Admin Manager

New Features

• The left navigation and header for Admin Manager have a new design. You can collapse the left navigation pane to increase space for the main panel and expand it to view page names:
  

  

• AutoSync environment admins can now update the AutoSync Snap Pack version on the AutoSync settings page. Learn more.
  

  

• The browser tab label now identifies Admin Manager and the name of the current page:
  

  

API Management

New Features

• The API metrics page is now in Monitor. You can view statistics and troubleshoot errors for your APIs.
  

  

• New public APIs for APIM to perform Git Integration Operations include: Create Branch, Create API version, Checkout branch, Pull latest changes, Git tag, Check Repo-Status, Discard changes, and Untrack a branch.

• New public API to perform a soft delete on APIs and API versions.

• A JWT Outbound policy is now available for Proxies. Your Snaplex must be on the May version.

Enhancements

• A new APIM function called request.roles enables API developers to invoke the list of existing roles for ease of use. Previously, you had to use IsUserinRole for each role type.

Fixed Issues

• The Request Transformer, Authorize by Role, and other Proxy-only policies now support Basic Auth for authentication. Previously, any policy executed after authentication would fail with Basic Authentication.

• Fixed an issue where the Client Throttle policy did not work when applied to an API Proxy running on a Snaplex with multiple nodes.

Deprecated

• The /wiki/spaces/AP/pages/337772580 is deprecated and the Apigee API Proxy selection will be removed from the Manager > Projects menu in July. This solution is no longer supported. We recommend that you use SnapLogic APIM for your integrations and APIs.

AutoSync

New Features

• Concatenate data from multiple columns or fields of a string type with the new Merge columns feature. Merging columns can simplify complex data structures and result in more efficient storage and querying. Learn more.
  

  

• Take advantage of new functionality and bug fixes by updating data pipelines to the latest AutoSync Snap Pack version. Environment admins can use the new Admin Manager AutoSync settings page to update multiple data pipelines at the same time. Learn more.
  

  

• New target endpoints: Databricks, Amazon S3, and Azure Data Lake Storage (ADLS) Gen2.

• You can now stop a data pipeline run mid-load. Because this can compromise target data integrity, we recommend stopping a data pipeline only when necessary. This might be, for example, with a data pipeline that takes a long time to load where you need to change the configuration or know that an issue with the source will result in errors. Learn more.

Enhancements

• The text on the edit dialog for data pipelines with a CSV file endpoint has been enhanced. Because CSV loads can’t be scheduled, the former AutoSynchronize tab is now Load type. The Append load type clarifies that when you select a file that was previously loaded, the entire file will be appended to the existing table.
  

  

• The AutoSync header has a new design:
  

  

• The browser tab now identifies AutoSync:
  

  

• The data preview table for transformations now shows 5 rows of data instead of 4.

Known Issues

• For the ServiceNow endpoint, string and decimal values load to the target using the target’s default precision. MySQL has a default precision of 255, so values with a precision of more than 255 will fail to load to MySQL.

• When creating a data pipeline with a CSV file source, the new Merge columns screen is skipped.

Fixed Issues

• The Infer data types for CSV files checkbox now only displays for the S3 and SFTP endpoints for CSV files, not for JSON files.

• For the ServiceNow endpoint, AutoSync ignores the maximum length defined by ServiceNow for string and decimal types and uses the default precision of the target.

Data Automation

Enhancements

• Enhanced the ELT Load Snap’s capabilities to allow loading flat and nested data sets (from canonical and non-canonical formats) from your Parquet files to the target tables in Snowflake. Learn more about the usage of this feature at Load data from Parquet files and in the following example pipelines:

  • Load data from a canonical format Parquet file

  • Load data from a non-canonical format Parquet file

Fixed Issue

• Fixed an issue where the ELT Merge Into Snap failed to load data into the Google BigQuery target table because of the error: Number of source column expressions in the input SQL is greater than the target table columns (even after altering the target table to match the source table columns). The resolution was to modify an SQL statement that was used internally to achieve the merge action.

Documentation Updates

Content Published on New Documentation Site

You can find the following content on docs.snaplogic.com:

• ML Analytics Snap Pack

• ML Natural Language Processing Snap Pack

Video

• Learn how to add a Response Transformer Policy to a Proxy.

Enhancements

• Content about configuring Azure Key Vault as a secrets manager has been validated and improved.

GenAI Builder

• Added the Anthropic Claude on AWS Messages Snap to the Amazon Bedrock Snap Pack.

Monitor

New Features

• The new Alert history displays all Snaplex node alerts to simplify review and analysis. You can tag events as open or closed to filter the list and you can download the log. Learn more.
  

  

• The Integrations catalog is available by request for Enterprise customers. Contact either adm-support@snaplogic.com or your CSM. The catalog displays an inventory of integrations for pipelines with an associated task in a production environment. The table includes metadata and you can add custom metadata columns and values. Learn more.
  

  

  
  For environments with thousands of pipelines, the Integrations catalog offers:

  • Improved visibility of Integrations as data assets

  • Better governance for DataOps

  • Faster impact analysis for System Architects

  • Easier identification of reusable assets for Developers to reduce integration development time

Enhancements

• Summary card counts on the Execution overview page now reflect any filters applied to the table in addition to the date range. For example, if two executions failed in the selected date range and were running on different Snaplexes, if you filter by one of the Snaplex names, the Failed card shows only one failure.

• The left navigation and header for Monitor have a new design. You can collapse the left navigation to increase space for the main panel:
  

  

• The browser tab now identifies Monitor and the name of the current page:
  

  

• The Activity Log includes usernames and Snaplex version updates when users initiate Snaplex version changes. This also applies to the Activity Log in the Classic Manager. Learn more

Known Issues

• With the Execution overview table grouped by Task or Pipeline, if you select a date range for more than one month's data, the execution table does not return results. The summary cards show the counts for executions by status.

• The summary card counts do not update correctly when you select the following advanced search filters for Invocation type:

  • Manual

  • Nested Pipeline

  • Error Pipeline

  • For Each

  • Regression Test

Platform

New Features

• The new public API for the Git Integration to discard local changes uses the project path and asset name to identify the assets to discard. Learn more.

• The new public Snaplex APIs enable administrators to:

  • Create a Groundplex

  • Update the Groundplex properties

  • Delete a Groundplex that does not have nodes

• The public API for users sends a notification email after a user is added successfully to the organization. Learn more.

Enhancements

• New support for AWS Secrets Manager to restrict secrets access by SnapLogic projects. To restrict access to specific projects, add role-based IAM aliases for them to a secrets-config.json file and restart your Groundplex nodes. Learn more.

• The header for Designer, Manager, and Dashboard has a new look:
  

  

• The user settings menu, which is available from a dropdown next to the user’s name, has a new design:

  • Settings are categorized as global or application-specific settings.

  • Non-admin users can’t change their username, they must ask an Environment admin to change it.
    The screenshot on the left shows the menu for Admin Manager, AutoSync, and Monitor. The screenshot on the right shows the menu for Dashboard and Manager:
    

    

• Browser tab labels now identify the app and page name:
  

• Open Telemetry reporting to third-party monitoring and logging tools now only sends messages labeled Error in severity level. Previously, messages marked Informational were also included. These Informational messages are still retained in the logs.

Fixed Issues

• Fixed an issue for the toPrecision() function that converts the data incorrectly when processing a whole number. When you call the function with arguments greater than 100, it results in an error.

Upcoming behavior change

Project-level pipeline execution behavior change

Currently, tasks stored in a Project Space of an Org can be executed by pipelines in a different Project Space or even from requests from a different Org. There is an upcoming fix to stop this behavior in the July release. After the fix is deployed, tasks must belong to the same Project Space as the pipeline Project Space they are called from to be successfully executed.

Known Issues

• When creating a Task and selecting the America/Mexico City time zone, the task runtime is off by one hour. Since Mexico City no longer observes daylight savings, the current implementation of CDT (aligned with CST) might assume that Mexico City changed time zones as it did in the past.
  Workaround: Upgrade your Snaplex to a JDK 11.0.18+ version.

Snaps

Upcoming

New Snaps

Introduced the following Snaps:

• Added MongoDB Execute and MongoDB - Atlas Vector Search Snaps to the MongoDB Snap Pack:

  • MongoDB Execute: Executes all Data Definition Language (DDL) and Data Manipulation Language (DML) commands on the MongoDB database.

  • MongoDB - Atlas Vector Search: Performs advanced vector-based queries, such as Similarity searches, Approximate Nearest Neighbor (ANN) queries, and Range queries on vector data stored in MongoDB Atlas.

• Added NetSuite Add List and NetSuite Async Add List Snaps to the NetSuite SOAP Snap Pack:

  • NetSuite Add List: Adds a list of records in bulk to NetSuite.

  • NetSuite Async Add List: Adds a list of records in NetSuite asynchronously.

• Added Parquet Parser and Parquet Formatter Snaps to the Transform Snap Pack:

  • Parquet Parser: Reads the binary Parquet data and writes document data to the output.

  • Parquet Formatter: Reads the document data and writes it to the output in binary Parquet format.

Enhancements

• The following Azure Active Directory-related Snaps and Accounts have been renamed because Microsoft has rebranded Azure Active Directory to Microsoft Entra ID.

• Enhanced the Google Directory Read Snap with an On empty result dropdown list that enables you to select the Snap behavior when no matching filter value is found.

• Enhanced the JSON Splitter Snap to capture metadata and lineage information from the input document.

• Kafka Snap Pack: Enhanced the following fields to support expressions through pipeline parameters:

  • Truststore Password

  • Keystore File Password

  • SSL Key Password

  • SASL Password or Secret

  • Registry Password or Secret

• Enhanced the MongoDB Update Snap with the Array Filters field, which enables you to use array filters in the update operation. Additionally, the Update Query field is now modified into a text box for visibility and usability of input queries.

• Increased the default Read timeout value for NetSuite REST Snaps from 60 to 300 seconds. This update supports longer-running operations and prevents early disconnections for complex data processing tasks.

• Updated the Delete Condition (Truncates a Table if empty) field in the following Database Delete Snaps to Delete condition (deletes all records from a table if left blank) to indicate that all entries will be deleted from the table when this field is blank, but no truncate operation is performed.

  • AlloyDB Delete

  • Azure Synapse SQL Delete

  • Databricks - Delete

  • MySQL - Delete

  • Oracle - Delete

  • PostgreSQL - Delete

  • Redshift - Delete

  • SAP HANA - Delete

  • Snowflake - Delete

  • SQL Server - Delete

  • Vertica - Delete

• Enhanced the HubSpot Snaps with the Retry policy field to handle retries for connection and error responses.

• Updated the following NetSuite SOAP Snap names for usability based on the functionality of the Snaps:

• Enhanced the S3 Select Snap to capture metadata and lineage information from the input document.

• Enhanced the Snowflake Bulk Load and Snowflake Insert Snaps to support iceberg tables. Iceberg tables partition and organize data across multiple nodes to distribute the workload and speed up data processing.

• Enhanced the SQL Server Snap Pack to support Geospatial data types (Geography and Geometry), which enables you to seamlessly insert, read, and select Geospatial data in the SQL Server environment.

• Upgraded Apache CXF from version 3.4.2 to 3.6.3 to prevent vulnerability issues for the following Snap Packs:

  • NetSuite SOAP

  • Workday

  • Xactly

  • Zuora

• Upgraded Spring dependencies to the latest supported Java 11 version for the MongoDB Snap Pack.

Behavior Change

MySQL - Insert Snap: With the 4.37 GA release, when you pass a Date/DateTime value for the Year data type column, the MySQL Insert Snap does not accept those values and fails with an error. This is the expected behavior of the Snap; we recommend you to pass only year values for the Year data type. Previously, when you passed a Date/DateTime value for the Year data type column, the Insert Snap inserted the value as a Year data type.

Fixed Issues

• Fixed an SMB (server message block) connectivity issue in the Binary Snap Pack, where the incorrect name provided by the SMB client did not match the Windows cluster virtual name, affecting the SPN (service principal name) connection.

• Fixed a performance issue in the Copy Item Snap, which previously resulted in a nameAlreadyExists error when the Snap retried because of a connection reset.

• Fixed an issue with the OpenAPI Snap where the upstream Mapper Snap did not display the full schema objects in the Target Schema.

• Fixed an issue with the Oracle - Bulk Load Snap that intermittently displayed a null pointer exception.

• Fixed the following issues with the SQL Server Snap Pack:

  • Snaps displayed an error when numeric values with scale 0 were inserted into the database with DECIMAL or NUMERIC type columns instead of inserting the value with decimal digits truncated.

  • Snaps displayed an error when fractional numeric values (numbers with a decimal point) were inserted into the database with an INT type column, instead of inserting the values with decimal digits truncated.

Known Issues

• The SQL Server - Bulk Load Snap returns 0 rows copied for tables containing spatial-type columns.

Deprecated

• The Azure Data Lake Accounts have been removed from the Binary and Hadoop Snap Packs because Microsoft retired the Azure Data Lake Storage Gen1 protocol on February 29, 2024. We recommend replacing your existing Azure Data Lake Accounts (in Binary or Hadoop Snap Packs) with other Azure Accounts.

SnapGPT

• Improved readability of Pipeline analysis summary and SnapGPT Analyze Pipeline Reports displays.

• Each SnapGPT completion displays feedback icons and a copy icon in the following UI contexts:

  • Generate pipeline: Copies the pipeline to be pasted into a Designer canvas.

  • Describe pipeline: Copies the summary and pastes it as markdown formatted text.

  • Question and answer: Copies the SnapGPT response to the question and pastes it as markdown formatted text.

  • Analyze pipeline: Copies the pipeline analysis and pastes it as JSON.

  • Error: Copies error information.