Highlights

The SnapLogic® August 2023 Release includes the following:

Admin Manager

• New interface to manage users, groups, service accounts, and roles

API Management

• Additional insights into Ultra Task requests on the APIM dashboard

AutoSync

• New source endpoints: Google Sheets, Snowflake, and Google BigQuery

• Propagate field and column data type changes to the destination

• New Append load type for CSV files

• Updated experience for sharing data pipelines

Platform

• Support for GitLabs

• App gateway for connectivity for on-premises Git servers

• Secrets Cache for Secrets Management

• Support for linking Git repositories with Project Spaces

• Support for Azure Key Vault using managed identity-based authentication

• LDAP authentication on HashiCorp Vaults

Snaps

• Introduced the following new Snap Packs:

  • Microsoft Dynamics 365 Business Central

  • Google Analytics 4

• Introduced the following Snaps:

  • GeoJSON Parser

  • Google BigQuery Upsert (Streaming)

  • HDFS Delete

  • PGP Sign

  • SAP SuccessFactors Read Advanced

  • WKT Parser

• SnapGPT (Early Access)

  You can now build your pipelines using simple text prompts.

August UAT Delta

About Upgrading to this Release

Dot Releases - Snaps

Admin Manager

Admin Manager is a new interface for administrators with an updated user experience based on feedback from you. The classic IIP Manager will be phased out when all settings become available in Admin Manager.

Open Admin Manager from the application menu:

Open

This release introduces a subset of features. Manage users, groups, service accounts, and roles from Admin Manager. Learn more.

Open

Features

Admin Manager introduces new concepts and terminology:

• Orgs are called environments, and an Org admin is called an Environment admin.

• Roles are used instead of groups to differentiate admins from regular users.

• Groups provide a way to share assets such as AutoSync data pipelines, AutoSync credentials, and IIP Account assets.

Known Issues

• On the Roles screen, the count of users for each role does not match the number shown on the details page. The details page opens when you click a role.

• When you delete a user and get a confirmation notice, the deleted user might display in the list for a short time while their assets are being deleted.

API Management

Enhancements

The APIM dashboard displays more data for Ultra requests. The dashboard displays the number of requests waiting in the queue for each minute, every FeedMaster node, and the total number of requests across all FeedMaster nodes. You can now view the average wait time, average processing time, and average total time for each Ultra Task.

The API dashboard also introduces a new Queue Size widget that displays the number of requests inline to be processed per time segment.

Known Issues

In the Response Mapper and Response Transformer policies, APIM functions display in the expressions dropdown menu but do not actually work.

AutoSync

Features

• AutoSync now supports the following endpoints:

  • Google Sheets as a source

  • Snowflake as a source

  • Google BigQuery as a source

• Detection of changes to source data types: Now AutoSync detects changes to source column or field data types and updates the destination table with a compatible type. Supported for Amazon Redshift, Google BigQuery, and Snowflake destinations. Learn more.

Enhancements

• You can now append data to a table created from a CSV file by splitting the file. This is useful, for example, when the data exceeds the 100 MB size limit or when a system periodically generates CSV files and you want them to load to the same table. Learn more.

• New experience for sharing data pipelines. Learn more.

• The Snowflake destination endpoint now uses internal Snowflake staging, which removes the need to have and configure a separate staging environment.

• The Amazon Redshift endpoint now supports using an S3 bucket for staging that is in a different AWS region. If the S3 bucket is in the same region as the Redshift instance, you can leave the region unspecified.

Known Issues

• When a data pipeline with Snowflake as a destination encounters an error, it stops loading. It should instead log the error record and continue.

Data Automation

Enhancement

• Upgraded the JDBC driver support for Snowflake to snowflake-jdbc-3.13.33.jar.

• Updated the ELT Snowflake Account to append or replace (if it exists) the connection parameter application in its JDBC URL with the value SnapLogic_iPaaS.

  • This change complies with Snowflake’s latest Connection Identifier standard.

Monitor

Enhancements

• The Studio dashboard displays nodes that are missing a heartbeat in a gray color and saves the details for a period of 45 days. Learn more.

Platform

Features

• Introducing Secrets Cache support for Secrets Management that minimizes latency resulting from fluctuating requests to all the Supported Secrets Management providers.

• Support for Git Lab Integration with your Org.

• Connectivity to on-premises GitHub Enterprise Servers (GHES). You can now configure the new app gateway to connect your Groundplex to your on-premises GHES host. Previously, connectivity required IP filtering on requests to GHES from the SnapLogic Control Plane or endpoint restrictions, where a limited subset of API endpoints was available.

• Support for Secrets Management using Azure Key Vault. In this release, we support only:

  • Secrets stored in the Azure Key Vault – and not keys and certificates.

  • Managed identity-based authentication on the Azure Active Directory (Azure AD): System-assigned identities and User-assigned identities.

• Snaplex node crashes are now captured as events in the Activity Log.

• Git repositories now link with Project Spaces. All Git operations supported in projects are also supported at the Project Space level.

• When you associate a repository with a Project Space, all the Git folders are brought in as Projects, and any assets at the root level are placed in a logical directory labeled as root. This label does not interfere with the corresponding Git folder structure. Learn more.
  
  

• You can add each individual untracked asset to the repository for tracking or remove one from the repository using the Remove from Repository option on that asset’s name.
  
  

Enhancements

• The IIP toolbar no longer collapses and the Export Pipeline icon has been updated. The following shows the Designer toolbar:

• The Import a Pipeline icon has been updated:

• LDAP authentication to HashiCorp Vaults. This enhancement might affect existing accounts based on HashiCorp Vaults. Refer to the following breaking change and the action needed.

Known Issues

• The Runtime Archive does not display content.

• The API response for Snaplex Configuration REST API gives the incorrect Snaplex version to download. Instead of the recommended Snaplex version, the response gets the latest version.
  Workaround: Use the Snaplex dialog Downloads tab for the recommended version.

• In GitHub, you cannot create a branch doing a Git checkout from the repository at the Project Space level.

• The Git status Updated does not work at the Project space level for all Git repos.

• In GitHub, when a root folder is already untracked in a Project Space after doing the checkout, the root (1) folder is not created.

• For GitLab integration, when updating the Git assets with the status Tracked Modified locally and remotely, a conflict error message displays.

• For GitLab integration, while creating a Project from a Git checkout, an error message displays.

• For GitLab integration, doing a Git Repository Checkout with Hard Reset results in an internal server error.

• If a self-managed GitLab server does not have GitLab as part of the hostname, Status, Pull, Commit, Create Branch, and Create Tag commands fail.

Snaps

Upcoming

New Snaps

• Introduced the Microsoft Dynamics 365 Business Central Snap Pack:

  • Dynamics 365 Business Central Create: Creates user records in the Business Central application for a specific entity.

  • Dynamics 365 Business Central Delete: Deletes user data from a specific entity.

  • Dynamics 365 Business Central Read: Reads user details for a specific entity.

  • Dynamics 365 Business Central Search: Searches for the required data for a specific entity

  • Dynamics 365 Business Central Update: Updates the user records for a specific entity.

  • Dynamics 365 Business Central OAuth2 Account: Connects the Snap with the Microsoft Dynamics 365 Business Central application that requires OAuth2 authorization.

  • Dynamics 365 Business Central Access Token Account: Connects the Business Central Snaps with data sources that dynamically use the Access Token.

• Introduced the Google Analytics 4 Snap Pack:

  • Create Realtime Report: Creates a customized report of your Google Analytics real-time event data.

  • Create Report: Creates a customized report of your Google Analytics event data.

  • Google Service Account: Authenticates and authorizes applications or services to access Google APIs.

• Introduced the following Snaps:

  • Google BigQuery Upsert (Streaming): This Google BigQuery Snap enables you to perform bulk update and insert operations into a BigQuery table from existing tables or any input data stream.

  • HDFS Delete: This Hadoop Snap provides a streamlined and efficient method for deleting files from Azure Data Lake Storage.

  • PGP Sign: This Binary Snap enables you to use PGP to sign a file digitally without encrypting the file.

  • SuccessFactors Read Advanced: This SAP SuccessFactors Snap enables you to read objects recursively for objects containing links to traverse through the tree structure.

  • Transform Snap Pack:

    • GeoJSON Parser: Parses geospatial data from binary data input and outputs the contents as a GeoJSON document downstream.

    • WKT Parser: Parses geospatial data from binary data input and outputs the contents as a WKT (Well-known Text) document downstream.

Enhancements

• All the database Execute Snaps now include a new Query type field. When Auto is selected, the Snap automatically determines the query type.

• Added new SFTP (Secure File Transfer Protocol) settings in the global.properties file:

  • sftpSessionTimeoutSec

  • cacheSftpSessions

• Anaplan

  • The Anaplan Action Snap now supports Data errors/warnings for better output handling. The reason for the process failure also displays now.

  • All the Anaplan Snaps now display the status code as part of the error message if the error status code is available as part of Anaplan API response.

• The Aggregate, Group By Fields, Group By N, Join, Sort, and Unique Snaps from the Transform Snap Pack now include new fields to improve memory management.

• The Amazon S3 Snaps automatically detect the Maximum session duration value for the Cross-Account IAM role (1 through 12 hours). The Snaps round down the value to the nearest hour. So, if the Snap administrator sets the Maximum session duration at 3 hours and 45 minutes, the Snaps read it as 3 hours. The Snaps also refresh the session before the session expires. However, the automatic session refresh does not support very large file uploads or downloads that takes longer than the maximum session duration.

• The Decrypt Field and Encrypt Field Snaps from the Transform Snap Pack now support CTR (Counter mode) for the AES (Advanced Encryption Standard) block cipher algorithm.

• The Decrypt Field Snap now supports the decryption of various encrypted fields by providing a valid decryption key.

• The Deduplicate Snap now manages memory efficiently and eliminates out-of-memory crashes using the following fields:

  • Minimum memory (MB)

  • Minimum free disk space (MB)

• The pagination feature implemented for the ABFS protocol in the Hadoop Directory Browser Snap enables the Snap to return all the output documents as expected.

• The SSH fields in the Generic Database account are now expression enabled.

• The LDAP Search and LDAP Lookup Snaps now support a string representation for binary attributes with the Include string representation for binary attributes checkbox.

• Microsoft Dynamics 365 for Sales Snap Pack:

  • The Dynamics 365 for Sales Snaps includes a new Enable continue on error checkbox to allow the Snap to continue executing the remaining batch requests, even if the previous request does not execute.

  • The Dynamics365 for Sales Update Snap performs the update operation as expected when the Batch size is more than 1, and the Snap successfully executes when the batch size is 1.

• The NetSuite Snap Pack now supports the Customer-Subsidiary and the Vendor-Subsidiary relationship Object types.
  Note: NetSuite's application requires the Oneworld extension for using these subsidiary relationships, and Snap integration must use WSDL 2019_1 or later versions.

• The PGP Encrypt Snap now includes an Encryption key ID field, which allows you to specify the key ID for encrypting the data. If you do not specify an encryption key ID, the Snap uses the first key in the public key (master key).

• The Redshift-Bulk Load and Redshift-Bulk Upsert Snaps now support expression enablers for the Additional options field, which enables you to use parameters.

• The Snowflake - Bulk Load and Snowflake - Bulk Upsert Snaps are enhanced with the Flush chunk size (in bytes) field. When using internal staging, data is written to a temporary chunk file. When the file exceeds the specified size, it is copied to the Snowflake stage and deleted.

• SQL Server Snap Pack:

  • This Snap Pack is upgraded to the Microsoft JDBC Driver 12.2.0.jre11 for SQL Server. Existing pipelines that do not use the default driver (bundled with the SQL Server Snap Pack) are not impacted. If you want to use NTLM or other authentications with the latest default driver (where the DLL file is mandatory), then you must copy the DLL file to the Snaplex path. Learn more about the SQL Server driver upgrade: FAQ for Upgrading to SQL Server JDBC Driver 12.2.0.

  • The SQL Server Stored Procedure and SQL Server Table List Snaps now support the retry functionality.

Fixed Issues

• Fixed the pagination issue for the Deals Entity in the HubSpot Read Snap.

• Fixed an issue in the SAP S/4HANA Create Snap where a few fields were missing in the schema population in the Mapper Snap.

• Fixed the encoding issue in the SAP S/4HANA Update Snap.

• Fixed an issue with the Send Email Snap where attachments larger than 3 MB were not getting attached to the sent email messages.

Behavior Changes

• Starting from version 433patches21482, the PGP Encrypt Snap does not support encryption with an expired key. To ensure continued support for encryption, we highly recommend that you renew your PGP key before it expires or extend the expiration date of your PGP key.

• Starting from version 433patches22343, the Vertica-Bulk Load Snap displays no output in the output view when the Snap receives no input.

• With the 433patches21307, pagination evaluation in the HTTP Client Snap occurs before sending the output to the output view. Therefore, if snap.out.totalCount is part of the Has next or any other pagination properties, the Snap will not include the count in the output corresponding to the request that produced the output document.

Known Issues

• For the Binary and Email Snap Packs, SFTP connections might experience connectivity issues because of the removal of some deprecated algorithms. To add these connections back, refer to Configuration Settings for Snaps for details.

• If you use the PostgreSQL driver (org.postgresql.Driver) with the Redshift Snap Pack, it could result in errors if the data type provided to the Snap does not match the data type in the Redshift table schema. Either use the Redshift driver (com.amazon.redshift.jdbc42.Driver) or use the correct data type in the input document to resolve these errors.

• The Send Email Snap might fail because of a 429 error response from the Graph API endpoint.

• For Ultra Tasks using a continuous data flow design for polling with a Pipeline Execute Snap, the parent pipeline does not receive the document from the child pipeline, and the polling message is not generated for downstream Snaps.

• The GeoJSON Parser and WKT Parser Snaps accept only application/zip or application/gml+xml; charset=utf-8 as a file input stream.
  However, the output content type from the File Reader Snap is binary/octet when the input file is read from SLDB.

  Workaround: To read an input file (ZIP or GML) from SLDB, use the Mapper Snap to change the content type from binary/octet to one of the following:

  • ZIP file: Change the content type to application/zip .

  • GML file: Change the content type to application/gml+xml; charset=utf-8.

• When you provide a custom JAR file in the Oracle Thin Account or the Oracle Thin Dynamic Account, the Oracle Stored Procedure Snap fails with the following errors:
  "com.snaplogic.snap.api.SnapDataException: object is not an instance of declaring class"
"Class oracle.sql.BLOB cannot be cast to class java.sql.Struct" (For BLOB data type)
  "class oracle.sql.CLOB cannot be cast to class java.sql.Struct" (For CLOB data type)

  Workaround: Remove the custom jar file from the account and retry.

Deprecated

The Microsoft Dynamics AX Snap Pack is deprecated. Contact support@snaplogic.com if your existing pipelines use Snaps from the deprecated Snap Pack.

Custom Snaps

Enhanced the following Custom Snap documents by adding missing information, improving clarity in field descriptions, and implementing the latest UI changes:

• Amazon ElastiCache Snap Pack

• AWS EC2 Utilities Snap Pack

• EDIFACT Snap Pack

SnapGPT

Jul 26, 2023: Launched SnapGPT, an AI-driven platform that lets you generate your pipelines using simple text prompts. Learn more.

Documentation Enhancements

• API Management VI - Create and Approve an Application and Subscription: In this final API Management video, learn how to manage users and API subscriptions for your Org’s Developer Portal. To learn more about API Management, watch the APIM-I, APIM-II, APIM-III, APIM-IV, and APIM-V videos in this series, in which we covered how to create, publish, and manage APIs and proxies in the API Manager and on the Developer Portal.

• Enhanced the Hyper Lib Setup and Tabcmd installation pages in the Tableau Snap Pack to enable you to perform tasks such as publishing workbooks and data sources, scheduling extract refreshes, creating users and groups, managing permissions, and interacting with Tableau Server or Tableau Online without relying on the Tableau user interface.

• Enhanced the Installing a Groundplex on a Docker Container page to include instructions to create a Docker image from a Groundplex.

• Updated the Snap Pack Versions by Release page for better readability and usability.