May 2021, 4.25 Release Notes

May 2021, 4.25 Release Notes

In this article

Key New Features in SnapLogic

Platform:

  • Support for elastic scaling in Kubernetes
  • Enhanced memory and resource management
  • Anomaly detection and active Pipeline analysis
  • Additional notifications via Slack

API Management:

  • Support for creating APIs from Swagger 2.0 specifications

  • Support for API versioning 

ELT for Cloud Data Platforms:

  • New! Launching Fast Data Loader
  • Introducing two new Snaps -- ELT Execute and ELT SCD2
  • Support for Snowflake that is hosted on Azure or Google Cloud Platform (GCP)

  • Support for ELT operations on data from Google Cloud Storage into Snowflake

  • Automatic download of JDBC Drivers for Snowflake, Redshift, and Azure Synapse 
  • Support for additional Window functions: Covariance, Correlation, and Linear Regression Functions

New Snaps:

  • Twilio Snap Pack

  • NetSuite Call RESTlet Snap
  • SAP RFC Listener Snap


Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the main-9727 - Patch 1 version at 9 p.m. PT on June 12, 2021. Orgs migrated to the 4.25 GA version prior to the auto-upgrade are not impacted. 
  • To use the new SnapLogic features and Snaps in the 4.25 release, ensure that your Snaplex nodes are upgraded to the recommended version.
  • This 4.25 release includes the patches deployed to SnapLogic after the 4.24 release, as listed in the 4.24 Dot Releases section. 
  • We remind you to add the IP addresses given in the SnapLogic allowlist article to your network.


4.25 Dot Releases

Platform

Date of UpdateSnaplex BuildUpdates
main-9988 - 4.25 Patch 3 
(Recommended)
  • Fixed an issue where Tasks with the + symbol in their names did not complete execution when activated.
  • Fixed an issue where service directory errors caused log files to increase in size.
  • Fixed an issue where memory usage for a Snaplex node increased despite a lack of Pipeline activity, unless restarted.
  • Fixed an issue where some Groundplex nodes experienced unexpectedly high CPU utilization.
  • Fixed an issue where Scheduled Tasks were not scheduled correctly because of daylight savings time.
  • Fixed an issue where some OAuth 2.0 endpoints did not support additional properties sent in the client body with credentials.
N/A

Control Plane update:

  • Fixed an issue where the public API for Pipeline runtime collection returned less runtimes than expected.
  • Fixed an issue where Task notifications were not taking effect despite being created successfully.
  • Fixed an issue in Designer where suggested Snaps for ELT Pipelines included standard-mode Snaps, even though they are not supported.
 main-9825 - 4.25 Patch 2 (Restricted)This is a restricted Snaplex build. For more information, contact support@snaplogic.com.
 main-9727 - Patch 1 
  • Fixed an issue where some recurring Scheduled Tasks caused duplicate Pipeline executions.
  • Fixed an issue where the Do not start a new execution if one is already active option in Scheduled Tasks was not enabled.
  • Fixed an issue with the OAuth 2.0 implementation for some Snap accounts by adding support to send client credentials as a Basic Auth header.
  • Fixed an issue with Snaps that allocate Snap metrics, where over time a memory leak occurred, requiring the reboot of the Snaplex node associated with the Pipeline.
 N/A

Control Plane update:

  • Fixed a caching issue with the UI.

  • Fixed an issue with the Project Migration API by restoring the original functionality of the duplicate_check parameter.
  • Fixed an issue where after overwriting a file in SnapLogic Manager successfully, the changes were not reflected in the actual file and the data remains unchanged when the file was reopened.

UI

Date of UpdateUpdate
 
  • Fixed an issue where opening the document preview for the second input of an Open API Snap caused the UI to freeze.
  • Fixed an issue where Pipeline Parameters were not displayed in the dialog window for new Triggered Tasks.
 
  • Fixed an issue with account validation where the Snaplex dropdown list Manager was empty when the Project Space or project name had special characters that required encoding.
  • Fixed an issue on the Create API form where re-uploading an API specification and clicking Cancel causes an unexpected error message.
  • Fixed an issue where the Task dialog window did not display the When Task Has options if you entered recipients in the Slack Notification field but left the Email Recipients field empty.
  • Fixed an issue where the Version Details page in the new API Manager console did not indicate on which server the API version was deployed.
  • Fixed an issue with the Dashboard > Snaplex tab where after selecting a Snaplex, all graphs displayed data for a week instead of the default time period (one hour).
  • Fixed an issue where the Triggered Task Details page did not display the Download the CA certificate option.
  • Fixed an issue where a Snap in the New Form UI did not display the default value for a new field in the Settings dialog window.
  • Fixed an issue where settings and recipients entered in the Task execution duration - time limit notification would not be recorded after reopening the completed notification.
  • Fixed an issue where the Mapper Snap Settings dialog window displayed the Missing property value error message twice after you deleted an existing expressions field.
  • Fixed an issue where the red arrow error indication was displayed in the password complexity tooltip.f
  • Fixed an issue with eXtreme Pipelines where the Disable Snap option (which did not function) was removed from the Snap Settings dialog window.
  • Fixed an issue where copying and pasting a Snap with a mouse caused an error, even after clearing the browser cache.
  • Fixed an issue where in some account types the Create Account dialog window did not display the username for the account.
  • Fixed an issue with the Select a Snaplex for validation dialog window where the Snaplex dropdown list displayed only Snaplex instances from the global shared folder.

Standard-mode Snaps

Loading

Data Automation

Loading


4.25 UAT Delta

 Click here to expand...

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT 1: Friday, April 23, 2021 – Snaplex version: 4.25_rc-9551 - 4.25 GA (Recommended)

  • UAT 2: Friday, April 30, 2021 
  • GA: Saturday, May 08, 2021 (9 p.m. PT)  

The SnapLogic GA release build on the production server is also deployed on the UAT server.

UAT 2 Fixes

Platform

Design-first API Management:

  • Fixed an issue with the API specification that fails to import while updating a Pipeline.
  • Fixed an issue where a user with full access to the SnapLogic Platform is unable to create an API.

UI

  • Fixed an issue with the new Task dialog window where you cannot assign another Snaplex to the task when the original Snaplex is deleted.
  • Fixed an issue with Scheduled Tasks in Manager that opens the Task Update dialog in the old UI, instead of the new UI.

Snaps


Platform 

Snaplex-based Scheduler

The Snaplex-based Scheduler will be the default scheduling mechanism for all Scheduled Tasks in the 4.26 release (August 2021). The new scheduling mechanism improves the timeliness and reliability of Scheduled Task executions, and we recommend that you enable the Snaplex-based Scheduler on your Orgs.

New Features

  • API Management - Design-First: Introduced an API Management section in Manager where you can create an API based on an Open API Specification (OAS) 2.0. As an Org admin, you can also manage APIs in the new API Manager console. The API Manager - /apim space supports the following activities:

  • Support for elastic scaling in Kubernetes: Introduced autoscaling for your Snaplex instances that are running in a Kubernetes environment. This implementation replaces the previous Kubernetes-based deployment of a Snaplex. Starting with the 4.25 release (May 2021), you can set up your Snaplex helm chart to initiate the scaling up and down of your node resources through Kubernetes orchestration. This implementation requires the Prometheus adapter and Grafana analytics.

  • Anomaly detection and active Pipeline analysis: Introduced two alert notifications for tracking Task executions that take a longer time than usual. You can define a hard limit either in minutes or by the percentage based on the last 30-day historical executions.


  • Enhanced memory and resource management to limit memory growth on Snaplex nodes. If the memory usage exceeds 95% of the configured maximum heap limit, the Pipelines that are causing the high memory utilization are terminated to prevent the Snaplex node from failing with an OutOfMemory error.

Enhancements to Existing Features

  • Snaplex monitoringAdded support for granularity in the reporting of data spikes in your Pipeline executions in the widgets on the Snaplex Dashboard Wall. You can now select Pipelines within a 15-minute window (indicated by the vertical line) and correlate active Pipelines with high resource utilization.


  • Additional notifications via Slack: Enhanced support for in-platform notifications. You can now add Slack recipients (channels and direct users) to Org-wide Snaplex notifications and Task notifications.
     
  • Asset Palette for APIs: Support for searching for Pipelines that are used by APIs that are created in the new API Management console.


  • Progressive loading of the Asset tree in Manager: Enhanced the search-and-browse experience in Manager. The assets are retrieved as needed, thus reducing latency.

  • UI refresh of Task dialog windows: Redesigned dialog windows for Triggered, Ultra, and Scheduled Tasks, giving the displays the same appearance as that of the New Form UI. As an enhancement, you can now select the Task type when opening the dialog window from Designer.

  • UI refresh of the Account dialog windows: Updated the UI and enhanced the support for creating accounts in Manager. Previously, in the 4.24 release and earlier, the option to create or modify an account using the New Form UI was available only in Designer.

Known Issues

  • When you migrate a project more than once, the existing assets are overwritten. You might have to associate your Tasks with the correct Pipelines after the migration. Typically, even after multiple migrations, account references remain intact in the Pipeline, but we recommend that you verify all references before starting your Pipeline executions. This behavior is a result of the duplicate_flag parameter being disabled.

  • When a start time of a Scheduled Task using the Snaplex-based scheduler is modified, the configured start time is not used to trigger the Pipeline execution.

  • In some scenarios with the Snaplex-based Scheduler, multiple executions of a Scheduled Task occur when only one instance is set to execute.

  • When calling a child Pipeline from a Groundplex to Cloudplex or vice versa, the Pipeline Execute Snap hangs, causing the parent Pipeline not to complete the execution.

  • The New Form UI does not show the default value for a custom property when editing an existing instance of a Snap.

  • In Manager, the Task Details page for some Triggered Tasks does not display the Download CA Certificate option.

  • When creating iterative Scheduled Tasks in time zone regions that have Daylight Savings, we recommend that you select the Cron Job option. See best practices for details.

Behavior Change

Retired the legacy UI: The legacy UI for Snaps is retired. Starting in the 4.25 release (May 2021), the option to deselect the New Form UI in the User Settings dialog window is no longer available. All Snaps can be viewed only in the new UI.


Standard-mode Snaps

New Snaps

  • NetSuite Call RESTletThis Snap provides the functionality of the NetSuite RESTlet framework that supports the HTTP methods (GET, DELETE, POST, PUT) and MIME types. You can call RESTlet scripts through token-based authentication.
  • SAP RFC Listener: This Snap facilitates Remote Function Calls (RFCs) between two SAP systems. The Snap simulates an RFC server and listens to the incoming RFC calls infinitely. Subsequently, the Snap generates output documents from the import parameters or triggers a child Pipeline. Business applications can use the RFC mechanism to communicate and exchange information (in predefined formats) with other systems.
  • Twilio Snap Pack: This Snap Pack enables you to perform various communication functions programmatically, such as making phone calls and sending text messages using Twilio's Web Service APIs. This Snap Pack has the following Snaps:

Updated Snap Packs

  • Binary Snap Pack:
    • Enhanced the Binary Snap Pack by adding an account Two-Factor Auth Account that supports two layers of authentication for Secure File Transfer Protocol (SFTP). The two secure layers of verification to grant access include the Basic Auth (username/password) and the SSH Key (username/private key) methods.
    • Enhanced the S3 File Writer Snap with the following abilities:
      • Use an MD5 checksum that checks for data integrity and corruption while uploading the file.
      • Write S3 objects with Access Control List (ACL) details in a single API call.

        Behavior Change

        In the 4.23 release, the S3 File Writer Snap used to write an S3 file and then set ACL if the ACL property was not empty. If the file write was successful and the ACL setting failed, both the output document in the output view and the error document in the error view was displayed. This issue has been resolved in the 4.24 release by enabling the Snap to configure ACL while writing the fileTherefore, the Snap produces an error document in the error view if the ACL configuration fails, irrespective of whether the file write action passes or fails.

  • Box Snap Pack:
    • Enhanced the Snaps in this Snap Pack with the Retry mechanism that includes the following settings:
      • Number of retries: Specifies the number of attempts the snap should make to perform the selected operation in case of connection failure or timeout.
      • Retry interval (seconds)Specifies the minimum number of seconds the Snap must wait before each retry attempt.
    • Enhanced the output view of the Box Directory Browser Snap to display the error message, Error retrieving contents of the folder with id:<Folder ID> instead of No folder exists with <Folder ID> when you enter an invalid folder ID.

  • DynamoDB Snap PackEnhanced the DynamoDB Account to support the Expression Enabler for account properties and creation of dynamic accounts.

  • Flow Snap PackEnhanced the HTTP Router Snap to support Ultra Pipelines.

  • Google Directory Snap Pack: Enhanced the Google Directory Insert and Google Directory Update Snaps to support custom schema attributes. You can fetch these custom fields and their values in a user profile by adding them as custom schema attributes to upstream Snaps.

  • Hadoop Snap Pack: Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
    • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
    • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.

  • Microsoft OneDrive Snap PackEnhanced the Update Share Permissions Snap with the Share Type and Expires fields that enable you to update the share permissions on your OneDrive items shared using the Invite or Link option. If the Share type is link, you can specify the expiry date and time for the URL that is used to share permissions.

  • Microsoft Power BI Snap Pack: Enhanced the following Snaps in the Microsoft Power BI Snap Pack by supporting data flows with two options, Dataflow and Dataflow Storage Accounts.
  • Salesforce Snap Pack:

    • Enhanced the pass-through behavior in Bulk API mode for Salesforce Write Snaps to include the input document in the original key when the batch size is set to 1. When the batch size is greater than 1, the input document is not included in the original key. This enhancement impacts the following Snaps:

    • Enhanced the Salesforce Upsert Snap to support CSV payload using the new field Bulk Content Type in Bulk API mode. The Salesforce API has recently introduced support for CSV payloads. In the 4.24 release and earlier, this Snap supported only the XML data type; starting from the 4.25 release, you can select CSV as the Bulk Content Type drop-down list. The default content type is set to XML to support backward compatibility. This enhancement also addresses the null values issue in Bulk API mode.
    • Enhanced the Snaps in the Salesforce Snap Pack by updating the field labels to the title case. If you have existing Pipelines that use Salesforce Snaps that include the field names in error messages, you must update those Pipelines to use the new field labels.

  • Snowflake Snap Pack:

    • Enhanced the Snowflake - Bulk Load Snap to enable data transformation using the new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. It also enables you to transform data types. This Snap supports CSV and JSON file formats for data transformation.

  • Transform Snap PackEnhanced the Group By N Snap with the following settings: 

    • Memory Sensitivity: The Snap's response to the memory changes. 
    • Group Size: The maximum number of input documents to be grouped into a single output document.
    • Min Group Size: The minimum number of input documents to be grouped into a single output document.
    • Flush Timeout: The number of seconds the Snap must wait before processing the records even if the incoming messages do not reach the specified Group Size.

Known Issues

Prior 4.24 Snap Dot Releases that are Merged into 4.25 GA

 Click here to expand...

Loading

Data Automation

SnapLogic's data automation solution speeds up the identification and integration of new data sources, and the migration of data from legacy systems. The solution can automatically detect duplicate, erroneous, or missing data, and identify structures and formats that do not match the data model. Data automation can accelerate the loading and transformation of your data into the data warehouse, speeding up the data-to-decisions process.

New Features in ELT Snap Pack

  • Starting with the 4.25 release, SnapLogic has now certified the ELT Snap Pack to work with Snowflake hosted on Google Cloud Platform (GCP) as the target database, in addition to the other flavors of Snowflake hosted on AWS and Microsoft Azure
  • Introduced the /wiki/spaces/AP/pages/1798014474 Snap to enable you to run DML, DDL, and DCL SQL queries in Snowflake in Snowflake, Redshift, and Azure Synapse.
  • Introduced the ELT SCD2 Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases—Snowflake, Redshift, and Azure Synapse.
  • Enhanced the ELT Database Account to introduce:
    • Support for Google Cloud Storage as a storage location (source) in addition to AWS S3 and Azure Data Lake Storage (ADLS) when your target database is Snowflake.
    • Automatic download of the JDBC driver required for the selected Database Type using the new Download JDBC Driver Automatically check box.

Supported JDBC JAR Versions

ELT Snaps automatically use a corresponding JDBC JAR file to connect to your target database and to perform the load and transform operations

DatabaseCertified JDBC JAR File
Azure Synapsemssql-jdbc-9.2.1.jre8.jar
Redshiftredshift-jdbc42-2.0.0.2.jar
Snowflakesnowflake-jdbc-3.13.1.jar

Using Alternate JDBC JAR File Versions

Though we recommend you to use the above JAR file versions, you can choose to use a different version, based on your environment. 

Updated ELT Snaps

  • Enhanced the ELT Load Snap to prevent changes to existing tables during Pipeline validation. If you set the Load Action as Drop and Create table, and the target table does not exist, the Snap creates a new (empty) target table based on the schema specified in its settings.
  • Enhanced the ELT Window Functions Snap to support Covariance, Correlation, and Linear Regression Functions on Snowflake, Redshift, and Azure Synapse databases. The Snap uses function-specific query re-writes to support these functions on Redshift and Azure Synapse databases.
  • Enhanced the ELT Merge Into and ELT Insert-Select Snaps to support up to one output view, and added the Get Preview Data check box to these Snaps. You can now connect downstream ELT Snaps to these Snaps.

Updated eXtreme Snap Packs

  • Introduced the SCD2 - Spark SQL 2.x Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases in the eXtreme mode.
  • Upgraded the /wiki/spaces/AP/pages/410189849to support Spark 3.0.1 on the following cloud platform versionsEnsure that you update your Snaplex to 4.25 GA before running your existing eXtreme Pipelines.
    • Amazon EMR 6.2.0 (Hadoop distribution: Amazon)
    • Azure Databricks 7.5

Known Issues

  • When ELT and Spark SQL 2.x Snap account credentials—such as user names, passwords, client secrets, auth codes and tokens, secret keys, and keystores—are auto-filled using the Google Chrome browser, the accounts, and hence the Pipelines fail. This is because the browser overwrites the field values with its own encrypted values that the SnapLogic Platform cannot read. SnapLogic recommends that you do not auto-save your Snap account credentials in the Chrome browser.
    Ensure that you delete any credentials that the browser has already saved for elastic.snaplogic.comand then perform ONE of the following actions:
    • Option 1: Click  that appears in the address bar after you submit your login credentials at elastic.snaplogic.comand then click Never
    • Option 2: Disable the Offer to save Passwords option at chrome://settings/passwords while working with your SnapLogic Pipelines. If you disable this option, your Chrome browser will not remember your passwords on any other website.
  • For Redshift and Azure Synapse databases, the ELT Load Snap displays JSON values for fields with datatypes Date and Time when validating the Pipeline.

    SnowflakeRedshiftAzure Synapse
    ”time_data” : “09:23:56”,”time_data” : {“time”: “09:23:56”},”time_data” : {“time”: “09:23:56”},
    ”date_data” : “2020-12-20”,”date_data” : “2020-12-20”,”date_data” : {“date”: “2020-12-20”},

    However, the ELT Load Snap outputs the correct date and time values when executing the Pipeline. 

    To prevent these values from showing up in JSON format in the previews and ELT Transform Snap, you can cast the time or date values to a VARCHAR type in your custom query as follows: 
    CAST (<time_column_type_name> AS VARCHAR(<size_based_on_precision_required>)).
    Example: CAST (time_data AS VARCHAR(20)).

  • ELT Transform Snap displays incorrect data types (string instead of the actual data type) for column names populated in its Input schema section.

New! Introducing Fast Data Loader (FDL) 

Ideal for business/data analysts, FDL is a data ingestion tool that simplifies the tasks of replicating and synchronizing data from the following source endpoints:

  • Coupa

  • SQL Server

  • Oracle

  • Salesforce

  • ServiceNow

  • Weather (pre-configured endpoint)

  • Covid (pre-configured endpoint)

    You can integrate your data source with the following destination data warehouses:

  • Redshift
  • SAP DWC
  • Snowflake

To gain your complimentary access to FDL, sign-up on the registration page. For more information, see the FDL user guide.  

Prior 4.24 eXtreme Dot Releases that are Merged into 4.25 GA

 Click here to expand...

Loading