August 2022 Release Notes

Key New Features

Platform:

  • Support for GitHub Enterprise Server.

  • Secrets Management using HashiCorp Vault to store Account credentials.

  • When setting permissions for a Project Space, Org admins can propagate permissions instead of setting them individually on every child Project.

AutoPrep:

  • This application enables you to profile your structured and unstructured data for analysis, reporting, and machine learning. AutoPrep is a new Snap in the Transform Snap Pack.

API Management:

  • In the API Dashboard, the Top Consumer by Requests widget is now available to display metrics about consumers of your APIs.

  • SSO authentication now works in the Developer Portal.

Studio:

  • View mean execution time for the last 1000 requests processed on FeedMaster nodes.

Snap Packs with New Snaps:

  • Amazon S3 Snap Pack

  • Databricks Snap Pack

  • Google BigQuery

  • Google Sheets

Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the recommended version at 9 p.m. PT on Saturday, Oct 1, 2022. Orgs migrated to the current GA version prior to the auto-upgrade are not impacted. 

  • To use the new SnapLogic features and Snaps in the current GA release, ensure that your Snaplex nodes are upgraded to the recommended version.

  • This release includes stable versions of the monthly releases and patches deployed to SnapLogic after the current GA release, as listed in the Dot Releases sections of the corresponding Release Notes. 

August UAT Delta

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT: Friday, Aug 5, 2022– Snaplex version: main-13224 - 4.30 UAT

  • GA: Thursday, Aug 11, 2022 Snaplex version: main-13269 - 4.30 GA

The SnapLogic GA release build on the production server is also deployed on the UAT server.

Dot Releases

Platform

Date of Update

Snaplex Build

Update

Date of Update

Snaplex Build

Update

Aug 22, 2022

main-13393 4.30 Patch 1 (Recommended)

Fixed an issue with runtime metrics that caused slow performance. We recommend that you use this version when you upgrade your Snaplexes to the August release.

Snaps

Snap Pack

Date of Update

Snap Pack Version

Updates

Snap Pack

Date of Update

Snap Pack Version

Updates

Oracle

Sep 7, 2022

430patches17658

The Oracle-Stored Procedure Snap now takes less time to execute when calling a stored procedure, because the Snap queries the metadata as required, thereby optimizing the Snap's performance.

Snowflake

Sep 2, 2022

4309atches17748

Fixes in the Snowflake Bulk Load Snap:

  • The Snap now routes the 403-Access Denied error to the error view (when the error view is enabled) when you do not have DeleteObject permission to delete the temporary files in the Amazon S3 bucket.

  • The Snap now checks for the expiration time when it receives a new access token for refreshing the token in a Snowflake OAuth2 account.

PostgreSQL

Sep 1, 2022

430patches17700

The PostgreSQL - Bulk Load Snap can now process the records with more than 16 KB in the document without encountering BufferOverflowException because the default value of 16 KB for byte buffer size is now removed.

Azure Active Directory

Aug 30, 2022

430patches17462

The List Entries, Get Group Members and Get Role Members Snaps now includes the Display Properties field that allows you to enter the user or group properties that you want to view in the output.

REST

Aug 30, 2022

430patches17684

The REST AWS Sig v4 Account is now enhanced with the AWS Region and Service Name fields to support Virtual Private Cloud (VPC) endpoint.

AWS Region: The AWS region that the account uses to construct the authorization header.

Service Name: The Service name that the account uses to construct the authorization header by extracting the values from the Service URL.

Workday

Aug 30, 2022

430patches17587

The performance of the Workday Read Snap is improved to reduce the execution time.

Microsoft Dynamics 365

Aug 29, 2022

430patches17665

The Microsoft Dynamics 365 for Sales Read Snap can now read up to five million records without hanging.

Transform

Aug 26, 2022

430patches17643

The CSV Parser and CSV Formatter Snaps now support either \ or \\ for a single backslash delimiter which were failing earlier.

Amazon S3

Aug 25, 2022

430patches17354

The KMS Region field in the S3 Account now suggests the regions on clicking the suggestion icon.

Snowflake

Aug 25, 2022

430patches17377

  • The following issues are fixed with the Snowflake - Bulk Load Snap:

    • The Snap successfully loads the data even when you deselect the Preserve case- sensitivity checkbox and the source and target column names are similar, but with different casing.

    • When the Snap has two input views, it creates the target table from the metadata provided by the second input view and loads the data into the target table from the first input view.

  • The Snowflake Select Snap now validates successfully even when you set the Table Name field with an expression.

SQL Server

Aug 25, 2022

430patches17064

The SQL Server - Select Snap now includes the Convert rowversion datatype to integer checkbox to support the ROWVERSION datatype. Select this checkbox to display the Rowversion datatype as integers in the output.

Transform

Aug 23, 2022

430patches17589

Now the CSV Formatter Snap does not hang when running in specific situations involving multibyte characters in a long field. If you notice the Snap stopping indefinitely, we recommend that you update to the 430patches17589 version and restart your Snaplex.

Email

Aug 22, 2022

430patches17403

The Email Account now does not display a null pointer exception when the Mail Properties fieldset is left blank as this is not a mandatory field. Earlier, the account displayed a null pointer exception when there were null values in this fieldset.

Eloqua

Aug 22, 2022

430patches17270

The Custom Object Metadata ID field in the Eloqua Create Snap allows you to specify an ID for creating custom object data. This conditional field appears only when you select Custom Object Data as the Core object type.

Dynamic 365 for Sales

Aug 22, 2022

430patches17054

The Dynamic 365 for Sales Update and Dynamic 365 for Sales Upsert Snaps now support expressions for the Key Columns field when you pass the alternate keys from Pipeline parameters and upstream Snaps.

Redshift

Aug 19, 2022

430patches17189

RabbitMQ

Aug 19, 2022

430patches17462

The RabbitMQ Account now successfully validates even when the port is out of range.

SQL Server

Aug 19, 2022

430patches17368

The SQL Server Execute Snap containing an explicit SQL transaction now works as expected when using a default JDBC driver.

Coupa

Aug 19, 2022

430patches17137

The Coupa Read Snap now reads all the records (using the Next Request) even when the page has fewer records, until the zero record count.

MongoDB

Aug 18, 2022

430patches17472

The MongoDB Account with Encryption type set to TLS/SSL does not fail with the URL cannot be null error.

SOAP

Aug 18, 2022

430patches17325

  • The patch 429patches17186, previously deployed for the SOAP Execute Snap caused an error (the failure of the getOrderStatus operation) even when the WSDL is valid. With the latest patch, this Snap generates the customized envelope without errors when you upload a valid WSDL.

  • The SOAP Execute Snap does not throw a null pointer exception when using the Import Schema http://schemas.xmlsoap.org/soap/encoding/.

MySQL

Aug 17, 2022

430patches17387

  • The following issues are fixed in the MySQL Bulk Load Snap:

    • Writing the first batch of records to a temporary file after the Snap fails. Now the Snap write to the error view as is the expected behavior.

    • Temporary files generation when record does not contain a table column. Now, the Snaplex removes the temporary files created along with the directory after the Pipeline execution.

SAP

Aug 17, 2022

430patches17344

The SAP Execute Snap now sends the actual content of the incoming byte arrays (in binary types like XString) instead of their transformed string values.

Binary

Aug 12, 2022

430patches17292

The Directory Browser Snap no longer fails with a null pointer exception when connecting to the SFTP server containing a port number.

The Binary Snap Pack is deployed as both the latest and stable distribution. We recommend that you use this version for your Org when using the recommended Snaplex version (main-13269 - 4.30 GA).

SnapLogic Studio (Preview)

Enhancements

Execution Time for Requests

FeedMaster node details include the mean execution time for the last 1000 processed requests for active Ultra Tasks:

Behavior Change

The Snaplex type label for Customer managed was changed to Self managed. From the System overview you can click a Snaplex to view its details, as shown below:

Snaplex Type

Patterns Catalog

The following Patterns were added to the Studio Pattern Catalog:

  • Convert Salesforce Lead to Account, Contact, and Opportunity

  • Create Box Folders for ServiceNow Service Requests

  • Create Microsoft Dynamics CRM Opportunity and Workday Invoice if Sale Occurs

  • Create Request Ticket in ServiceNow and Send Email

  • Create Request Ticket in ServiceNow When Employee Added in Workday

  • Creating a ServiceNow Incident From a Jira Issue

  • Create Users in SnapLogic With Excel and REST

  • Create Users in SnapLogic With JSON Generator and REST

  • Create Users in ServiceNow Through Workday Approval Process

  • CSV to Workday Tenant


Platform

The SnapLogic platform goes live on Cloudflare on Sep 8, 2022. When launched, the SnapLogic UI will automatically redirect to the specific Cloudflare CDN addresses. This change enhances the SnapLogic platform’s response time, security, and performance. To ensure continued access to the SnapLogic UI, only customers who restrict outbound IP addresses to a predefined list of IP addresses must extend their allowlist to add all the specific IP addresses. Learn more to understand how this impacts your organization.

New Features

GitHub Enterprise Server

The SnapLogic Git Integration supports GitHub Enterprise Server (GHES). From the SnapLogic user interface, you can use GHES to track SnapLogic Projects, Files, Pipelines, Tasks, and Accounts. Similar to the Git Integration for GitHub, the GHES Integration enables you to manage your Project Assets across development, test, and production Orgs, to achieve continuous integration and development (CI/CD). Support for GHES Integration is a premium feature that requires a subscription.

Using GHES Git Integration requires the following:

  1. A subscription to the Git Integration for GHES.

  2. Configuration of GHES.

  3. Configuration of the network to support communication between SnapLogic and GHES.

  4. Configuration of SnapLogic Org settings.

When the configuration is finished, each user logs in to SnapLogic to authorize and authenticate with their GHES account. After a Project is associated with a repository, the controls to check out, pull, add, delete, and commit are available for Project Assets. Learn more about the Git Integration and Enabling the Git Integration for GHES.

Secrets Management - HashiCorp Vault

Many organizations must comply with internal or external regulations for controlling access to sensitive information such as account credentials. The SnapLogic Secrets Management feature allows organizations to use a third-party secrets manager to store credentials. Instead of entering credentials directly in SnapLogic Accounts and relying on SnapLogic to encrypt them, the Accounts contain only the information necessary to access the secrets manager. During validation and execution, Pipelines obtain the credentials directly from the secrets manager. With Secrets Management, secrets are never stored by the SnapLogic control plane or by Snaplex nodes. With the August 2022 Release, SnapLogic supports the HashiCorp Vault Cloud, Enterprise, and Open Source editions. Secrets Management is a premium feature that requires a subscription.

The high-level tasks required to use Secrets Management include the following:

  1. Use of self-managed Snaplexes (Groundplexes) and a subscription to Secrets Management.

  2. An administrator of the secrets manager must configure the storage for endpoint credentials, create authentication roles and access permissions, and generate secrets.

  3. A Snaplex administrator must configure the Groundplex nodes with the token and the information required to communicate with the secrets manager.

  4. In SnapLogic, a Pipeline designer or Org admin must configure the dynamic accounts to authenticate with the secrets manager.

Only SnapLogic dynamic accounts support Secrets Management.

Learn more about using HashiCorp Vault with SnapLogic.

Enhancements

Propagate Project Space Permissions

When adding user or group permissions for a Project Space, Org admins can specify that those permissions should apply to all Projects. After you add propagated permissions, the user or group displays only in the Permissions dialog for the shared folder, not for the individual projects. Although they don’t show in the Permissions dialog for individual projects, they have the permission granted at the Project Space level:

Known Issues

The following issues affect customers using the Git Integration:

  • An error can occur when users who have not authenticated before try to connect to GitHub.

  • When you pull from GitHub, the message does not display all filenames correctly.

  • A commit to GHES can fail with the message, “Work Tree clean, nothing to commit.”

API Management

New Features

  • New widget in the API Dashboard. In the API tab of the Dashboard, you can add the new Top Consumer by Requests widget to display metrics about consumers of your APIs. Learn more about API Insights.

  • SSO authentication for the Developer Portal. If a user logs in to SnapLogic through an SSO provider and the user navigates from the Portal Manager to the Org’s Developer Portal, the user is taken directly to the Developer Portal landing page without having to log in again. Learn more about setting up SSO in SnapLogic.


Snaps

Read about the Snap features and fixes deployed in the prior 4.29 Dot Releases which are part of the current GA version.

The Snap Catalog has a new category for custom Snaps developed by the SnapLogic Professional Services team. For more information on the custom Snaps and how you can subscribe to them, contact your SnapLogic Customer Service Manager.

New Snaps

Amazon S3: Introduces the S3 Snap Pack that allows you to manage objects in S3. This Snap Pack contains the following Snaps:

  • S3 Browser: Lists the attributes of S3 objects in the specified bucket matching the prefix.

  • S3 Copy: Sends a copy request to the S3 service to copy an S3 object from a source bucket to a target bucket.

  • S3 Delete: Removes an object from the specified bucket.

  • S3 Download: Downloads S3 objects from the S3 bucket.

  • S3 Upload: Uploads binary data to S3 objects.

  • S3 Presigned: Generates a presigned URL in the output document to access an S3 object.

  • S3 Account: Enables you to access resources on the S3 bucket.

Databricks: A new Snap Pack for Databricks Lakehouse Platform (Databricks or DLP) introduces the following Snaps:

Google BigQuery: Introduces the following Snaps and accounts to manage datasets in Google BigQuery (GBQ):

Google Sheets: Introduces the following Snaps to subscribe to and unsubscribe from Google spreadsheets.

Transform Snap Pack: Introducing a data preparation application within a Snap that offers a new user experience.

  • AutoPrep Snap: Enables you to profile a data set before forwarding the data for further processing. You can flatten structured data, include and exclude data fields, and change data types through menu-like displays, making Pipeline building easier.

Enhancements

  • All Snaps with the JSON type output preview include the Render whitespace checkbox, which enables or disables the rendering of whitespace in the output. The Snap renders blank spaces as a dot “.” and tab spaces as an underscore “_”.

  • Dynamic Account types across all the Snap Packs support Secrets Management, a SnapLogic add-on that allows you to store endpoint credentials in a third-party secrets manager. Orgs using Secrets Management provide the information necessary to retrieve the secrets in expression-enabled dynamic Account fields. During validation and execution, Pipelines obtain the credentials directly from the Secrets Manager.

  • Azure Active Directory

  • Binary

    • The File Operation Snap supports moving data from a local node to an Azure blob through the AZ Copy utility.

    • The Azure Storage Account includes the Request Size (MB) field to set the buffer limit before writing to Azure storage to enhance the performance.

    • The SSH Auth Account supports dynamic values for the following fields that enable you to use Pipeline parameters:

      • Username

      • Private key

      • Key passphrase

  • Dynamo DB

    • The DynamoDB Account includes the AWS Security Token field to pass the security token when making calls using temporary credentials.

  • Email

    • The Email Sender Snap supports the file:/// protocol in the Attachments field.

  • Google BigQuery

    • The Location field in the BigQuery Execute Snap lists all the locations in the suggestions.

  • Google Directory

  • Google Sheets

    • The Allow empty rows checkbox in the Worksheet Reader Snap allows you to output empty rows. Select this checkbox to include empty rows between other rows that contain data.

    • The Worksheet Writer Snap supports expressions for the Starting Cell Reference field. This Snap also alerts you about the potential overwriting of existing data for the Starting Cell Reference field when writing data in the same range of cells.

  • Hadoop

  • JDBC

    • Enhanced the Generic Database account to support dynamic values for Account properties, Advanced properties, and URL properties.

  • Kafka

    • Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1.

    • Optimized the performance of the Snap with the support for LZ4 and ZSTD compression types of the messages produced.

    • The Kafka account that connects to the Kafka Schema Registry uses TLS v1.2 protocol instead of the older version of TLS, because it has security vulnerabilities.

  • MS Dynamics 365 for Sales

  • OpenAPI

    • The OpenAPI Snap supports external files referenced in the OpenAPI specification field and includes the following new fields:

      • The Pass through checkbox allows you to pass input data to the output under the original key.

      • The Trust all certificates checkbox allows you to use self-signed certificates.

      • The Enable URL Encoding checkbox allows automatic URL encoding of the request path.

  • Oracle

    • Enhanced the Oracle - Merge and Oracle - Update Snaps with the Session parameters fieldset to set the parameters for National Language Support (NLS).

    • Enhanced the Oracle - Stored Procedure Snap with a process that caches the fetched procedure metadata. Hence, the Snap processes the metadata calls only for the first few documents and does not process for subsequent documents, thereby improving the Snap's performance.

  • Postgres SQL

    • Enhanced the PostgreSQL Account and PostgreSQL Dynamic Account with SSH Tunneling configurations to encrypt the network connection between the client and the PostgreSQL Database server, thereby ensuring the secure network connection.

  • RabbitMQ

    • The RabbitMQ Consumer and RabbitMQ Producer Snaps can consume and produce messages, respectively, from the quorum queue by setting the Argument key field as x-queue-type and the Argument value as quorum to consume or publish messages from/to the quorum queue.

  • Redshift

    • The Redshift accounts support:

      • Expression enabler to pass values from Pipeline parameters.

      • Security Token for S3 bucket external staging.

  • REST

    • You can set cookie policy specifications using the Cookie Policy dropdown list in REST Snaps .

    • Support for auto-retries where the REST Snaps automatically retry when they encounter the 429 HTTP status code - Too Many Requests error.

    • The REST AWS Sig v4 Account supports:

      • Passing AWS Security Token Service (STS) temporary credentials in the Security Token field.

      • Expression values for Access-key ID and Secret Key fields.

    • Improved the performance of REST Snaps when the Snaps encounter Unauthorized (401) or Forbidden (403) errors even when the user credentials are available in the organization and the account type is OAuth2.

  • Salesforce

    • Updates in Salesforce Subscriber Snap:

      • CometD version upgrade from 5.0.9 to 7.0.6 (latest).

      • The simplified logging provides useful diagnostic information without logging any sensitive data.

      • The Snap asynchronously makes callbacks or listens to the CometD API to identify and respond to specific events.

      • The Snap processes the most recent message so that it can send that replay ID if there is a need to resubscribe. If the server responds with an invalid replay ID, the Snap automatically sets the Replay ID value to -2 to get all available messages.

    • You can select the Null Setting with Bulk API checkbox in the Salesforce Update, Salesforce Create, and Salesforce Upsert Snaps to format null values in the input document.


  • Snowflake

  • Transform

    • The Hide whitespace option in the CSV Generator and JSON Generator Snaps allows you to hide the rendering of whitespace as symbols (dot or underscore) in the output that you may have in the CSV or JSON input documents.

    • The Render whitespace checkbox in the Mapper Snap enables or disables the rendering of whitespace in the input document. When a value in the Expression field has blank spaces (leading, trailing, or spaces in the middle of a string), the spaces are rendered as symbols (dot “.” or underscore “_”) in the output on selecting this checkbox.

    • The Excel Parser Snap includes the Custom Locale dropdown list that allows selecting a user-defined locale for formatting numbers.

    • The Selected fields in the Pivot Snap allow you to define fields to be unpivoted so that the remaining fields are automatically pivoted.

    • The XML Generator Snap includes examples on how to escape single (') and double quotes (“) when used with elements or attributes.

  • The Snap Catalog has a new category for Custom Snaps developed by the SnapLogic Professional Services team. To subscribe to Custom Snaps, or if you have questions about Custom Snaps, contact your SnapLogic Customer Service Manager (CSM).

Deprecated Snaps

The Microsoft SOAP-based API (Organization Service) and the WS-Trust Security Protocol are being deprecated. Therefore, the Microsoft Dynamics CRM Snap Pack is no longer being updated and will be removed from the Snap Catalog in a future release. If you are having any issues with your existing Dynamics CRM Pipelines, contact support@snaplogic.com.

Known Issues

  • Recent changes in Kafka to enable idempotent writes by default may result in authorization errors when using the Kafka Snaps if your Kafka cluster version is lower than 2.8.

    To fix this issue, we recommend you take one of the following actions:

    • Upgrade your Kafka cluster to at least version 2.8.

    • Use Kafka’s ACL management to ensure that IdempotentWrite is allowed for the cluster.

Learn more in this community post.

  • When you add an input view to the Databricks - Delete Snap, ensure that you configure the Batch size as 1 in the Snap’s account configuration. For any other batch size, the Snap fails with the exception: Multi-batch parameter values are not supported for this query type.

  • [This issue is fixed in the July 2022 Release]: The Databricks - Unload Snap fails with the error: External source/target type is invalid when you attempt unloading data from your DLP instance to a DBFS location.


Data Automation

ELT Snap Pack

The following enhancements are deployed to the SnapLogic production environment as a stable version of the ELT Snap Pack in this release (4.30 GA). The latest versions have been deployed in either the June 2022 or July 2022 releases.

Enhancements

  • The ELT Load Snap can infer the schema from the source files located in S3 buckets and create, overwrite, and append the target table in your Redshift instance with the source data. It can infer the table schema from the data available in AVRO, CSV, JSON, ORC, and PARQUET files. Learn more at Automatic Schema Inference with ELT Load Snap.

  • The ELT Snap Pack supports the latest JDBC drivers across CDWs—Azure Synapse, BigQuery, DLP, Redshift, and Snowflake. Learn more in Configuring ELT Database Accounts or the respective Account page for the specific versions.

  • The Value List field in the ELT Pivot Snap provides dynamic suggestions.

  • The ELT Snowflake Account supports Key Pair Authentication on the cloud environment hosting your Snowflake instance.

  • You can configure S3 Bucket, Azure Storage, and DeltaLake Storage Gen2 mounts in the ELT DLP Account.

  • The end Snap SQL query in the ELT Insert Select Snap’s preview output displays the CREATE TABLE... or the DELETE/DROP TABLE statements to be run before the query that inserts/loads data into a new table in the Snowflake target CDW.

  • The ELT Load Snap now seamlessly reads from a CSV file in the S3 mount point on DBFS (DLP target instance) without causing the SQL exception—[Simba][SparkJDBCDriver](500051) ERROR processing query/statement.

  • The ELT Insert Select, ELT Merge Into, ELT Load, and the ELT SCD2 Snaps run successfully even when the specified target table does not exist. The Snaps now create a new target table if it does not exist during Pipeline validation.

The newly created table will not be dropped if a downstream Snap fails during validation.

  • The ELT SCD2 Snap:

    • On Pipeline validation, displays the final SQL query to be executed on the target CDW in its output preview.

    • Can replace an existing target table with a new table and load the SCD2 entries from the source table and files into it. Use the new Overwrite existing table option in the Target Table action field to perform this action.

  • Field names across the ELT Aggregate, ELT Cast Function, ELT String Function, and ELT Transform Snaps are normalized—to maintain consistency.

Known Issues