August 2022 Release Notes -- UAT version

Key New Features in SnapLogic

Studio:

  • View mean execution time for the last 1000 requests processed on FeedMaster nodes.

Platform:

  • Support for GitHub Enterprise Server (with a subscription).

  • Secrets Management using HashiCorp Vault to store Account credentials (with a subscription).

  • When setting permissions for a Project Space, Org admins can propagate permissions instead of setting them individually on every child Project.

API Management:

  • In the API Dashboard, the Top Consumer by Requests widget is now available to display metrics about consumers of your APIs.

Snaps:

  • Google BigQuery:

    • BigQuery Dataset Create

    • BigQuery Dataset Delete

    • BigQuery Dataset List

    • BigQuery Table Delete

    • BigQuery Table List

  • Google Sheets

    • Google Sheet Subscribe

    • Google Sheet Unsubscribe

  • Databricks Snap Pack

  • Amazon S3 Snap Pack

  • Transform Snap Pack

    • AutoPrep Snap: Organize and prune your structured and unstructured data for analysis, reporting, and machine learning with this easy application inside the Pipeline.

Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the recommended version at 9 p.m. PT on Thursday, Sep 10, 2022. Orgs migrated to the current GA version prior to the auto-upgrade are not impacted. 
    The Snaplex upgrade process also upgrades the Java version to the latest version. After the upgrade, the Java version of your Cloudplex nodes might not be the same as your FeedMaster nodes. However, this does not impact the operations of your SnapLogic instance. We will be fixing this issue in a subsequent release.

  • To use the new SnapLogic features and Snaps in the current GA release, ensure that your Snaplex nodes are upgraded to the recommended version.

  • This release includes stable versions of the monthly releases and patches deployed to SnapLogic after the current GA release, as listed in the Dot Releases sections of the corresponding Release Notes. 

4.30 UAT Delta

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT: Friday, Aug 5, 2022– Snaplex version: X.XX_rc-NNNNN (Recommended)

  • GA: Thursday, Aug 11, 2022

The SnapLogic GA release build on the production server is also deployed on the UAT server.

SnapLogic Studio (Preview)

Enhancements

Execution Time for Requests

FeedMaster node details include the mean execution time for the last 1000 processed requests for active Ultra Tasks:

Execution Time

Behavior Change

The Snaplex type label for Customer managed was changed to Self managed. From the System overview you can click a Snaplex to view its details, as shown below:

Patterns Catalog

The following Patterns were added to the Studio Pattern Catalog:

  • Convert Salesforce Lead to Account, Contact, and Opportunity (Optional)

  • Create Box Folders for ServiceNow Service Requests

  • Create Microsoft Dynamics CRM Opportunity and Workday Invoice if Sale Occurs

  • Create Request Ticket in ServiceNow and Send Email

  • Create Request Ticket in ServiceNow When Employee Added in Workday

  • Creating a ServiceNow Incident From a JIRA Issue

  • Creating Users Within SnapLogic (Excel + REST)

  • Creating Users Within SnapLogic (JSON Generator + REST)

  • Create Users Within ServiceNow Through Workday Approval Process

  • CSV to Workday Tenant

Known Issues

TBD


Platform

The SnapLogic platform goes live on Cloudflare on Sep 8, 2022. When launched, the SnapLogic UI will automatically redirect to the specific Cloudflare CDN addresses. This change enhances the SnapLogic platform’s response time, security, and performance. To ensure continued access to the SnapLogic UI, only customers who restrict outbound IP addresses to a predefined list of IP addresses must extend their allowlist to add all the specific IP addresses. Learn more to understand how this impacts your organization.

New Features

GitHub Enterprise Server

The SnapLogic Git Integration supports GitHub Enterprise Server (GHES). From the SnapLogic user interface, you can use GHES to track SnapLogic Projects, Files, Pipelines, Tasks, and Accounts. Similar to the Git Integration for GitHub, the GHES Integration enables you to manage your Project Assets across development, test, and production Orgs, to achieve continuous integration and development (CI/CD).

Using GHES Git Integration requires the following:

  1. A subscription to the Git Integration for GHES.

  2. Configuration of GHES.

  3. Configuration of the network to support communication between SnapLogic and GHES.

  4. Configuration of SnapLogic Org settings.

When the configuration is finished, each user logs in to SnapLogic to authorize and authenticate with their GHES account. After a Project is associated with a repository, the controls to check out, pull, add, delete, and commit are available for Project Assets. Learn more about the Git Integration and Enabling the Git Integration for GHES.

Secrets Management - HashiCorp Vault

Many organizations must comply with internal or external regulations for controlling access to sensitive information such as account credentials The SnapLogic Secrets Management feature allows organizations to use a third-party secrets manager to store credentials. Instead of entering credentials directly in SnapLogic Accounts and relying on SnapLogic to encrypt them, the Accounts contain only the information necessary to access the secrets manager. During validation and execution, Pipelines obtain the credentials directly from the secrets manager. With Secrets Management, secrets are never stored by the SnapLogic control plane or by Snaplex nodes. With the August 2022 Release, SnapLogic supports the HashiCorp Vault Cloud, Enterprise, and Open Source editions.

The high-level tasks required to use Secrets Management include the following:

  1. Use of self-managed Snaplexes (Groundplexes) and a subscription to Secrets Management.

  2. An administrator of the secrets manager must configure the storage for endpoint credentials, create authentication roles and access permissions, and generate secrets.

  3. A Snaplex administrator must configure the Groundplex nodes with the token and the information required to communicate with the secrets manager.

  4. In SnapLogic, a Pipeline designer or Org admin must configure the dynamic accounts to authenticate with the secrets manager.

Learn more about using HashiCorp Vault with SnapLogic.

Enhancements

Propagate Project Space Permissions

When adding user or group permissions for a Project Space, Org admins can specify that those permissions should apply to all Projects. After you add propagated permissions, the user or group displays only in the Permissions dialog for the shared folder, not for the individual projects. Although they don’t show in the Permissions dialog for individual projects, they have the permission granted at the Project Space level.

Known Issues

TBD

API Management

New Features

  • New widget in the API Dashboard. In the API tab of the Dashboard, you can add the new Top Consumer by Requests widget to display metrics about consumers of your APIs. Learn more about API Insights.

  • SSO authentication for the Developer Portal. If a user logs in to SnapLogic through an SSO provider and the user navigates from the Portal Manager to the Org’s Developer Portal, the user is taken directly to the Developer Portal landing page without having to log in again. Learn more about setting up SSO in SnapLogic.


Snaps

New Snaps

Amazon S3: Introduces the S3 Snap Pack that allows you to manage objects in S3. This Snap Pack contains the following Snaps:

  • S3 Browser: Lists the attributes of S3 objects in the specified bucket matching the prefix.

  • S3 Copy: Sends a copy request to the S3 service to copy an S3 object from a source bucket to a target bucket.

  • S3 Delete: Removes an object from the specified bucket.

  • S3 Download: Downloads S3 objects from the S3 bucket.

  • S3 Upload: Uploads binary data to S3 objects.

  • S3 Presigned: Generates a presigned URL in the output document to access an S3 object.

  • S3 Account: Enables you to access resources on the S3 bucket.

Databricks: A new Snap Pack for Databricks Lakehouse Platform (Databricks or DLP) introduces the following Snaps:

Google BigQuery: Introduces the following Snaps and accounts to manage datasets in Google BigQuery (GBQ):

Google Sheets: Introduces the following Snaps to subscribe to and unsubscribe from Google spreadsheets.

Transform Snap Pack: Introducing a data preparation application within a Snap that offers a new user experience.

  • AutoPrep Snap: Enables you to profile a data set before forwarding the data for further processing. You can flatten structured data, include and exclude data fields, and change data types through menu-like displays, making Pipeline building easier.

Enhancements

  • All Snaps with the JSON type output preview include the Render whitespace checkbox, which enables or disables the rendering of whitespace in the output. The Snap renders blank spaces as a dot “.” and tab spaces as an underscore “_”.

  • Dynamic Account types across all the Snap Packs support Secrets Management, a SnapLogic add-on that allows you to store endpoint credentials in a third-party secrets manager. Orgs using Secrets Management provide the information necessary to retrieve the secrets in expression-enabled dynamic Account fields. During validation and execution, Pipelines obtain the credentials directly from the Secrets Manager.

  • Azure Active Directory

  • Binary

    • The File Operation Snap supports moving data from a local node to an Azure blob through the AZ Copy utility.

    • The Azure Storage Account includes the Request Size (MB) field to set the buffer limit before writing to Azure storage to enhance the performance.

    • The SSH Auth Account supports dynamic values for the following fields that enable you to use Pipeline parameters:

      • Username

      • Private key

      • Key passphrase

  • Dynamo DB

    • The DynamoDB Account includes the AWS Security Token field to pass the security token when making calls using temporary credentials.

  • Email

    • The Email Sender Snap supports the file:/// protocol in the Attachments field.

  • Google BigQuery

    • The Location field in the BigQuery Execute Snap lists all the locations in the suggestions.

  • Google Directory

  • Google Sheets

    • The Allow empty rows checkbox in the Worksheet Reader Snap allows you to output empty rows. Select this checkbox to include empty rows between other rows that contain data.

    • The Worksheet Writer Snap supports expressions for the Starting Cell Reference field. This Snap also alerts you about the potential overwriting of existing data for the Starting Cell Reference field when writing data in the same range of cells.

  • Hadoop

  • JDBC

    • Enhanced the Generic Database account to support dynamic values for Account properties, Advanced properties, and URL properties.

  • Kafka

    • Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1.

    • Optimized the performance of the Snap with the support for LZ4 and ZSTD compression types of the messages produced.

    • The Kafka account that connects to the Kafka Schema Registry uses TLS v1.2 protocol instead of the older version of TLS, because it has security vulnerabilities.

  • MS Dynamics 365 for Sales

  • OpenAPI

    • The OpenAPI Snap supports external files referenced in the OpenAPI specification field and includes the following new fields:

      • The Pass through checkbox allows you to pass input data to the output under the original key.

      • The Trust all certificates checkbox allows you to use self-signed certificates.

      • The Enable URL Encoding checkbox allows automatic URL encoding of the request path.

  • Oracle

    • Enhanced the Oracle - Merge and Oracle - Update Snaps with the Session parameters fieldset to set the parameters for National Language Support (NLS).

    • Enhanced the Oracle - Stored Procedure Snap with a process that caches the fetched procedure metadata. Hence, the Snap processes the metadata calls only for the first few documents and does not process for subsequent documents, thereby improving the Snap's performance.

  • Postgres SQL

    • Enhanced the PostgreSQL Account and PostgreSQL Dynamic Account with SSH Tunneling configurations to encrypt the network connection between the client and the PostgreSQL Database server, thereby ensuring the secure network connection.

  • RabbitMQ

    • The RabbitMQ Consumer and RabbitMQ Producer Snaps can consume and produce messages, respectively, from the quorum queue by setting the Argument key field as x-queue-type and the Argument value as quorum to consume or publish messages from/to the quorum queue.

  • Redshift

    • The Redshift accounts support:

      • Expression enabler to pass values from Pipeline parameters.

      • Security Token for S3 bucket external staging.

  • REST

    • You can set cookie policy specifications using the Cookie Policy dropdown list in REST Snaps .

    • Support for auto-retries where the REST Snaps automatically retry when they encounter the 429 HTTP status code - Too Many Requests error.

    • The REST AWS Sig v4 Account supports:

      • Passing AWS Security Token Service (STS) temporary credentials in the Security Token field.

      • Expression values for Access-key ID and Secret Key fields.

    • Improved the performance of REST Snaps when the Snaps encounter Unauthorized (401) or Forbidden (403) errors even when the user credentials are available in the organization and the account type is OAuth2.

  • Salesforce

    • Updates in Salesforce Subscriber Snap:

      • CometD version upgrade from 5.0.9 to 7.0.6 (latest).

      • The simplified logging provides useful diagnostic information without logging any sensitive data.

      • The Snap asynchronously makes callbacks or listens to the CometD API to identify and respond to specific events.

      • The Snap processes the most recent message so that it can send that replay ID if there is a need to resubscribe. If the server responds with an invalid replay ID, the Snap automatically sets the Replay ID value to -2 to get all available messages.

    • You can select the Null Setting with Bulk API checkbox in the Salesforce Update, Salesforce Create, and Salesforce Upsert Snaps to format null values in the input document.

  • SharePoint Online

  • Snowflake

  • Transform

    • The Hide whitespace option in the CSV Generator and JSON Generator Snaps allows you to hide the rendering of whitespace as symbols (dot or underscore) in the output that you may have in the CSV or JSON input documents.

    • The Render whitespace checkbox in the Mapper Snap enables or disables the rendering of whitespace in the input document. When a value in the Expression field has blank spaces (leading, trailing, or spaces in the middle of a string), the spaces are rendered as symbols (dot “.” or underscore “_”) in the output on selecting this checkbox.

    • The Excel Parser Snap includes the Custom Locale dropdown list that allows selecting a user-defined locale for formatting numbers.

    • The Selected fields in the Pivot Snap allow you to define fields to be unpivoted so that the remaining fields are automatically pivoted.

    • The XML Generator Snap includes examples on how to escape single (') and double quotes (“) when used with elements or attributes.

  • The Snap Catalog has a new category for Custom Snaps developed by the SnapLogic Professional Services team. To subscribe to Custom Snaps, or if you have questions about Custom Snaps, contact your SnapLogic Customer Success Manager (CSM).

Deprecated Snaps

The Microsoft SOAP-based API (Organization Service) and the WS-Trust Security Protocol are being deprecated. Therefore, the Microsoft Dynamics CRM Snap Pack is no longer being updated and will be removed from the Snap Catalog in a future release. If you are having any issues with your existing Dynamics CRM Pipelines, contact support@snaplogic.com.

Known Issues

  • Recent changes in Kafka to enable idempotent writes by default may result in authorization errors when using the Kafka Snaps if your Kafka cluster version is lower than 2.8.

    To fix this issue, we recommend you take one of the following actions:

    • Upgrade your Kafka cluster to at least version 2.8.

    • Use Kafka’s ACL management to ensure that IdempotentWrite is allowed for the cluster.

Learn more in this community post.

  • When you add an input view to the Databricks - Delete Snap, ensure that you configure the Batch size as 1 in the Snap’s account configuration. For any other batch size, the Snap fails with the exception: Multi-batch parameter values are not supported for this query type.

  • [This issue is fixed in the July 2022 Release]: The Databricks - Unload Snap fails with the error: External source/target type is invalid when you attempt unloading data from your DLP instance to a DBFS location.


Data Automation

ELT Snap Pack

The following enhancements are deployed to the SnapLogic production environment as a stable version of the ELT Snap Pack in this release (4.30 GA). The latest versions have been deployed in either the June 2022 or July 2022 releases.

Enhancements

  • The ELT Load Snap can infer the schema from the source files located in S3 buckets and create, overwrite, and append the target table in your Redshift instance with the source data. It can infer the table schema from the data available in AVRO, CSV, JSON, ORC, and PARQUET files. Learn more at Automatic Schema Inference with ELT Load Snap.

  • The ELT Snap Pack supports the latest JDBC drivers across CDWs—Azure Synapse, BigQuery, DLP, Redshift, and Snowflake. Learn more in Configuring ELT Database Accounts or the respective Account page for the specific versions.

  • The Value List field in the ELT Pivot Snap provides dynamic suggestions.

  • The ELT Snowflake Account supports Key Pair Authentication on the cloud environment hosting your Snowflake instance.

  • You can configure S3 Bucket, Azure Storage, and DeltaLake Storage Gen2 mounts in the ELT DLP Account.

  • The end Snap SQL query in the ELT Insert Select Snap’s preview output displays the CREATE TABLE... or the DELETE/DROP TABLE statements to be run before the query that inserts/loads data into a new table in the Snowflake target CDW.

  • The ELT Load Snap now seamlessly reads from a CSV file in the S3 mount point on DBFS (DLP target instance) without causing the SQL exception—[Simba][SparkJDBCDriver](500051) ERROR processing query/statement.

  • The ELT Insert Select, ELT Merge Into, ELT Load, and the ELT SCD2 Snaps run successfully even when the specified target table does not exist. The Snaps now create a new target table if it does not exist during Pipeline validation.

  • The ELT SCD2 Snap:

    • On Pipeline validation, displays the final SQL query to be executed on the target CDW in its output preview.

    • Can replace an existing target table with a new table and load the SCD2 entries from the source table and files into it. Use the new Overwrite existing table option in the Target Table action field to perform this action.

  • Field names across the ELT Aggregate, ELT Cast Function, ELT String Function, and ELT Transform Snaps are normalized—to maintain consistency.

Known Issues