May 2021, 4.25 Release Notes
In this article
Key New Features in SnapLogic
Platform:
- Support for elastic scaling in Kubernetes
- Enhanced memory and resource management
- Anomaly detection and active Pipeline analysis
- Additional notifications via Slack
API Management:
Support for creating APIs from Swagger 2.0 specifications
- Support for API versioning
ELT for Cloud Data Platforms:
- New! Launching Fast Data Loader
- Introducing two new Snaps -- ELT Execute and ELT SCD2
Support for Snowflake that is hosted on Azure or Google Cloud Platform (GCP)
Support for ELT operations on data from Google Cloud Storage into Snowflake
- Automatic download of JDBC Drivers for Snowflake, Redshift, and Azure Synapse
- Support for additional Window functions: Covariance, Correlation, and Linear Regression Functions
New Snaps:
Twilio Snap Pack
- NetSuite Call RESTlet Snap
- SAP RFC Listener Snap
Important
- Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the main-9727 - Patch 1 version at 9 p.m. PT on June 12, 2021. Orgs migrated to the 4.25 GA version prior to the auto-upgrade are not impacted.
- To use the new SnapLogic features and Snaps in the 4.25 release, ensure that your Snaplex nodes are upgraded to the recommended version.
- This 4.25 release includes the patches deployed to SnapLogic after the 4.24 release, as listed in the 4.24 Dot Releases section.
- We remind you to add the IP addresses given in the SnapLogic allowlist article to your network.
4.25 Dot Releases
Platform
Date of Update | Snaplex Build | Updates |
---|---|---|
main-9988 - 4.25 Patch 3 (Recommended) |
| |
N/A | Control Plane update:
| |
main-9825 - 4.25 Patch 2 (Restricted) | This is a restricted Snaplex build. For more information, contact support@snaplogic.com. | |
main-9727 - Patch 1 |
| |
N/A | Control Plane update:
|
UI
Date of Update | Update |
---|---|
| |
|
Standard-mode Snaps
Snap Pack | Date of Update | Snap Pack Build Name | Updates |
---|---|---|---|
Box | 425patches10840 | Modified default values for the following fields in Box Read and Box Write Snaps:
For existing Pipelines which have the Enable Staging (deprecated field) checkbox selected, this Snap retries the connection once even if the Number of Retries is zero. | |
DynamoDB Flow JIRA Microsoft OneDrive Teams SAP | 425patches10571 |
| |
Amazon SQS Binary Coupa Exchange Online ML Data Preparation Slack SOAP Tableau Workday Prism Analytics | 425patches10994 |
| |
JDBC MySQL Oracle Redshift Teradata | 425patches11008 |
| |
/wiki/spaces/AP/pages/1439348 | 425patches10880 | Improved the Dynamics 365 for Sales Upsert Snap by caching the parameter lookup values. | |
Azure Active Directory | 425patches10868 | Enhanced the Create Entry and Update Entry Snaps to support Pipeline parameters and upstream values for the Attribute name setting under Attributes and the Snap Pack to support proxy authentication. | |
SQL Server | 425patches10797 | Fixed an issue with the SQL Server - Bulk Load Snap where the Snap fails when the login password contains a colon or a less than (<) symbol. | |
REST | 425patches10419 | Fixed a memory leak issue in the REST Snap Pack. | |
Anaplan | 425patches10360 | Added the following fields to the Anaplan Write Snap:
| |
Salesforce | 425patches10182 | Fixed an issue with the Salesforce Read Snap where the retry fields do not work as expected when a Pipeline has more than ten Snaps and improved the error messages in the Snap where it fails with a null pointer exception error when the account reference provided is invalid. | |
REST | 425patches9929 | Enhanced the REST Snap Pack to support Proxy Authentication. | |
Transform | 425patches10663 | Fixed an issue in the CSV Formatter Snap where even if the Ignore empty stream checkbox is not enabled, the Snap did not produce an empty binary stream output in case there is no input document. | |
Reltio | 425patches10410 | Fixed an issue with the Reltio Create Snap where the Snap processes a lesser number of documents while loading data into Reltio. | |
Transform | 425patches10152 | Enhanced the CSV Parser Snap with a new checkbox Preserve Surrounding Spaces that enables you to preserve the surrounding spaces for the values that are non-quoted. Replaced the Strict XSD output field with Map input to first repeating element in XSD field in the XML Formatter Snap. If selected, the Snap ignores the root element from the XSD file. | |
SOAP | 425patches10152 | Enhanced the SOAP Execute Snap with a new checkbox Escape special characters that enable the snap to escape XML special characters in XML template variable values when this Snap inserts these values into the Apache Velocity template. | |
Snowflake | 425patches10190 | Enhanced the Snowflake S3 Database and Snowflake S3 Dynamic accounts with a new field S3 AWS Token that allows you to connect to private and protected Amazon S3 buckets. | |
Confluent Kafka | 425patches10543 |
| |
OpenAPI | 425patches10417 | Enhanced the OpenAPI Snap with the following two fields:
| |
Salesforce | 425patches9609 | Enhanced the Salesforce Read Snap to allow you to add an optional second output view that exposes the schema of the target object as the output document. The output view of the Snap has a minimum of one output and a maximum of two outputs. | |
/wiki/spaces/AP/pages/1439348 | 425patches9813 | Fixed an issue with /wiki/spaces/AP/pages/1689750 Snap where the Snap returns duplicate records when reading results from multiple pages. | |
Binary | 425patches10217 |
| |
MS Dynamics 365 for Sales | 425patches10303 | Fixed an error in the Dynamics 365 for Sales Create Snap where the Snap does not send the original documents to the error view in the output when the batch size property is set to greater than 1. | |
ServiceNow | 425patches10014 | Enhanced the ServiceNow Query Snap with the retry mechanism that includes the following settings:
| |
Box | 425patches10072 | Modified default values of the following fields across the Box Snap Pack to trigger automatic retries by default, in case of connection failures or timeouts:
| |
Hadoop | 425patches9975 | Fixed a dependency issue in Hadoop Parquet Reader Snap while reading data from the AWS S3 folder. The issue is caused due to conflicting definitions for some of the AWS class dependencies in the classpath. | |
PostgreSQL | 425patches9879 | Enhanced the performance of PostgreSQL - Bulk Load Snap significantly. We expect the Snap to execute up to 3 times faster than the previous version for enterprise workloads. | |
Transform | 425patches9815 | Fixed a ClassCastException error in the Avro Parser Snap and handling of the map, fixed, enum, and bytes data types in the Avro Formatter and Avro Parser Snaps. | |
Binary | 425patches9581 | Fixed an issue with the PGP Decrypt Snap where the Snap writes an empty document to the output view when there is an error in the Snap. It will now decrypt the document and write it to the output view only if it succeeds. As of we have discovered an issue with this Binary patch affecting the PGP Decrypt Snap. In case your Pipelines use the PGP Decrypt Snap, we recommend you to NOT use this patch. Else, your Pipelines will likely fail. We're working on a fix and will be deploying a Snap patch soon. | |
Transform | 425patches9749 | Enhanced the XML Parser Snap to recognize input headers when defining the inbound schema. | |
Amazon SQS | 425patches9596 | Added the Security token field to the Amazon SQS Account. | |
REST | 425patches9684 |
| |
Binary | 425patches9564 | Enhanced the SFTP session in Binary Snaps to support the SFTP servers that close the SFTP sessions slowly. | |
Transform | 425patches9638 | Reverted the Join Snap to the 4.24 release behavior. This is in response to an issue encountered in the Join Snap in the 4.25 release version (main9554) that can result in incorrect outputs from all Join Types. 425patches9638 is the default version for both stable and latest Transform Snap Pack versions for orgs that are on the 4.25 release version. No action is required by customers to receive this update and no impact is anticipated. For more information, contact support@snaplogic.com. |
Data Automation
Snap Pack | Date of Update | Snap Pack Build Name | Updates |
---|---|---|---|
ELT | 425patches10017 |
| |
ELT | 425patches9725 |
|
4.25 UAT Delta
Platform
Snaplex-based Scheduler
The Snaplex-based Scheduler will be the default scheduling mechanism for all Scheduled Tasks in the 4.26 release (August 2021). The new scheduling mechanism improves the timeliness and reliability of Scheduled Task executions, and we recommend that you enable the Snaplex-based Scheduler on your Orgs.
New Features
- API Management - Design-First: Introduced an API Management section in Manager where you can create an API based on an Open API Specification (OAS) 2.0. As an Org admin, you can also manage APIs in the new API Manager console. The API Manager - /apim space supports the following activities:
- Create an API by uploading a Swagger 2.0 specification or by referencing it through a URL.
- Create multiple versions of your API and manage their assets.
- Add permissions to your APIs and versions. If you are an Org admin, you can set permissions at the API Manager - /apim space level.
- Apply API policies at the API and version levels.
- View details about APIs, Versions, and their Assets.
- Support for elastic scaling in Kubernetes: Introduced autoscaling for your Snaplex instances that are running in a Kubernetes environment. This implementation replaces the previous Kubernetes-based deployment of a Snaplex. Starting with the 4.25 release (May 2021), you can set up your Snaplex helm chart to initiate the scaling up and down of your node resources through Kubernetes orchestration. This implementation requires the Prometheus adapter and Grafana analytics.
- Anomaly detection and active Pipeline analysis: Introduced two alert notifications for tracking Task executions that take a longer time than usual. You can define a hard limit either in minutes or by the percentage based on the last 30-day historical executions.
- Enhanced memory and resource management to limit memory growth on Snaplex nodes. If the memory usage exceeds 95% of the configured maximum heap limit, the Pipelines that are causing the high memory utilization are terminated to prevent the Snaplex node from failing with an OutOfMemory error.
Enhancements to Existing Features
- Snaplex monitoring: Added support for granularity in the reporting of data spikes in your Pipeline executions in the widgets on the Snaplex Dashboard Wall. You can now select Pipelines within a 15-minute window (indicated by the vertical line) and correlate active Pipelines with high resource utilization.
- Additional notifications via Slack: Enhanced support for in-platform notifications. You can now add Slack recipients (channels and direct users) to Org-wide Snaplex notifications and Task notifications.
- Asset Palette for APIs: Support for searching for Pipelines that are used by APIs that are created in the new API Management console.
- Progressive loading of the Asset tree in Manager: Enhanced the search-and-browse experience in Manager. The assets are retrieved as needed, thus reducing latency.
- UI refresh of Task dialog windows: Redesigned dialog windows for Triggered, Ultra, and Scheduled Tasks, giving the displays the same appearance as that of the New Form UI. As an enhancement, you can now select the Task type when opening the dialog window from Designer.
- UI refresh of the Account dialog windows: Updated the UI and enhanced the support for creating accounts in Manager. Previously, in the 4.24 release and earlier, the option to create or modify an account using the New Form UI was available only in Designer.
Known Issues
- When you migrate a project more than once, the existing assets are overwritten. You might have to associate your Tasks with the correct Pipelines after the migration. Typically, even after multiple migrations, account references remain intact in the Pipeline, but we recommend that you verify all references before starting your Pipeline executions. This behavior is a result of the
duplicate_flag
parameter being disabled. - When a start time of a Scheduled Task using the Snaplex-based scheduler is modified, the configured start time is not used to trigger the Pipeline execution.
- In some scenarios with the Snaplex-based Scheduler, multiple executions of a Scheduled Task occur when only one instance is set to execute.
- When calling a child Pipeline from a Groundplex to Cloudplex or vice versa, the Pipeline Execute Snap hangs, causing the parent Pipeline not to complete the execution.
- The New Form UI does not show the default value for a custom property when editing an existing instance of a Snap.
- In Manager, the Task Details page for some Triggered Tasks does not display the Download CA Certificate option.
- When creating iterative Scheduled Tasks in time zone regions that have Daylight Savings, we recommend that you select the Cron Job option. See best practices for details.
Behavior Change
Retired the legacy UI: The legacy UI for Snaps is retired. Starting in the 4.25 release (May 2021), the option to deselect the New Form UI in the User Settings dialog window is no longer available. All Snaps can be viewed only in the new UI.
Standard-mode Snaps
New Snaps
- NetSuite Call RESTlet: This Snap provides the functionality of the NetSuite RESTlet framework that supports the HTTP methods (GET, DELETE, POST, PUT) and MIME types. You can call RESTlet scripts through token-based authentication.
- SAP RFC Listener: This Snap facilitates Remote Function Calls (RFCs) between two SAP systems. The Snap simulates an RFC server and listens to the incoming RFC calls infinitely. Subsequently, the Snap generates output documents from the import parameters or triggers a child Pipeline. Business applications can use the RFC mechanism to communicate and exchange information (in predefined formats) with other systems.
- Twilio Snap Pack: This Snap Pack enables you to perform various communication functions programmatically, such as making phone calls and sending text messages using Twilio's Web Service APIs. This Snap Pack has the following Snaps:
- Twilio Read Snap: Retrieves data resources from Twilio's REST APIs.
- Twilio Create Snap: Creates data objects in Twilio.
- Twilio Modify Snap: Modifies existing Twilio data objects.
- Twilio Delete Snap: Deletes existing data objects from Twilio.
Updated Snap Packs
- Binary Snap Pack:
- Enhanced the Binary Snap Pack by adding an account Two-Factor Auth Account that supports two layers of authentication for Secure File Transfer Protocol (SFTP). The two secure layers of verification to grant access include the Basic Auth (username/password) and the SSH Key (username/private key) methods.
- Enhanced the S3 File Writer Snap with the following abilities:
- Use an MD5 checksum that checks for data integrity and corruption while uploading the file.
Write S3 objects with Access Control List (ACL) details in a single API call.
Behavior Change
In the 4.23 release, the S3 File Writer Snap used to write an S3 file and then set ACL if the ACL property was not empty. If the file write was successful and the ACL setting failed, both the output document in the output view and the error document in the error view was displayed. This issue has been resolved in the 4.24 release by enabling the Snap to configure ACL while writing the file. Therefore, the Snap produces an error document in the error view if the ACL configuration fails, irrespective of whether the file write action passes or fails.
- Box Snap Pack:
- Enhanced the Snaps in this Snap Pack with the Retry mechanism that includes the following settings:
- Number of retries: Specifies the number of attempts the snap should make to perform the selected operation in case of connection failure or timeout.
- Retry interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
Enhanced the output view of the Box Directory Browser Snap to display the error message,
Error retrieving contents of the folder with id:<Folder ID>
instead ofNo folder exists with <Folder ID>
when you enter an invalid folder ID.
- Enhanced the Snaps in this Snap Pack with the Retry mechanism that includes the following settings:
- DynamoDB Snap Pack: Enhanced the DynamoDB Account to support the Expression Enabler for account properties and creation of dynamic accounts.
- Flow Snap Pack: Enhanced the HTTP Router Snap to support Ultra Pipelines.
- Google Directory Snap Pack: Enhanced the Google Directory Insert and Google Directory Update Snaps to support custom schema attributes. You can fetch these custom fields and their values in a user profile by adding them as custom schema attributes to upstream Snaps.
- Hadoop Snap Pack: Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
- Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
- Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
- Microsoft OneDrive Snap Pack: Enhanced the Update Share Permissions Snap with the Share Type and Expires fields that enable you to update the share permissions on your OneDrive items shared using the Invite or Link option. If the Share type is link, you can specify the expiry date and time for the URL that is used to share permissions.
- Microsoft Power BI Snap Pack: Enhanced the following Snaps in the Microsoft Power BI Snap Pack by supporting data flows with two options, Dataflow and Dataflow Storage Accounts.
Enhanced the pass-through behavior in Bulk API mode for Salesforce Write Snaps to include the input document in the
original
key when the batch size is set to 1. When the batch size is greater than 1, the input document is not included in theoriginal
key. This enhancement impacts the following Snaps:- Enhanced the Salesforce Upsert Snap to support CSV payload using the new field Bulk Content Type in Bulk API mode. The Salesforce API has recently introduced support for CSV payloads. In the 4.24 release and earlier, this Snap supported only the XML data type; starting from the 4.25 release, you can select CSV as the Bulk Content Type drop-down list. The default content type is set to XML to support backward compatibility. This enhancement also addresses the null values issue in Bulk API mode.
- Enhanced the Snaps in the Salesforce Snap Pack by updating the field labels to the title case. If you have existing Pipelines that use Salesforce Snaps that include the field names in error messages, you must update those Pipelines to use the new field labels.
- Enhanced the Snowflake - Bulk Load Snap to enable data transformation using the new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. It also enables you to transform data types. This Snap supports CSV and JSON file formats for data transformation.
- Enhanced the Snowflake - Bulk Load Snap to enable data transformation using the new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. It also enables you to transform data types. This Snap supports CSV and JSON file formats for data transformation.
Transform Snap Pack: Enhanced the Group By N Snap with the following settings:
- Memory Sensitivity: The Snap's response to the memory changes.
- Group Size: The maximum number of input documents to be grouped into a single output document.
- Min Group Size: The minimum number of input documents to be grouped into a single output document.
- Flush Timeout: The number of seconds the Snap must wait before processing the records even if the incoming messages do not reach the specified Group Size.
Known Issues
The Generic JDBC - Insert Snap in the JDBC Snap Pack might fail due to a connection issue. We recommend that you roll back to the main8614 version.
The Avro Parser Snap in the Transform Snap Pack might fail due to
ClassCastException
error. We recommend that you upgrade to the 425patches9815 version.
Prior 4.24 Snap Dot Releases that are Merged into 4.25 GA
Data Automation
SnapLogic's data automation solution speeds up the identification and integration of new data sources, and the migration of data from legacy systems. The solution can automatically detect duplicate, erroneous, or missing data, and identify structures and formats that do not match the data model. Data automation can accelerate the loading and transformation of your data into the data warehouse, speeding up the data-to-decisions process.
New Features in ELT Snap Pack
- Starting with the 4.25 release, SnapLogic has now certified the ELT Snap Pack to work with Snowflake hosted on Google Cloud Platform (GCP) as the target database, in addition to the other flavors of Snowflake hosted on AWS and Microsoft Azure.
- Introduced the /wiki/spaces/AP/pages/1798014474 Snap to enable you to run DML, DDL, and DCL SQL queries in Snowflake in Snowflake, Redshift, and Azure Synapse.
- Introduced the ELT SCD2 Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases—Snowflake, Redshift, and Azure Synapse.
- Enhanced the ELT Database Account to introduce:
- Support for Google Cloud Storage as a storage location (source) in addition to AWS S3 and Azure Data Lake Storage (ADLS) when your target database is Snowflake.
- Automatic download of the JDBC driver required for the selected Database Type using the new Download JDBC Driver Automatically check box.
Supported JDBC JAR Versions
ELT Snaps automatically use a corresponding JDBC JAR file to connect to your target database and to perform the load and transform operations.
Database | Certified JDBC JAR File |
---|---|
Azure Synapse | mssql-jdbc-9.2.1.jre8.jar |
Redshift | redshift-jdbc42-2.0.0.2.jar |
Snowflake | snowflake-jdbc-3.13.1.jar |
Using Alternate JDBC JAR File Versions
Though we recommend you to use the above JAR file versions, you can choose to use a different version, based on your environment.
Updated ELT Snaps
- Enhanced the ELT Load Snap to prevent changes to existing tables during Pipeline validation. If you set the Load Action as Drop and Create table, and the target table does not exist, the Snap creates a new (empty) target table based on the schema specified in its settings.
- Enhanced the ELT Window Functions Snap to support Covariance, Correlation, and Linear Regression Functions on Snowflake, Redshift, and Azure Synapse databases. The Snap uses function-specific query re-writes to support these functions on Redshift and Azure Synapse databases.
- Enhanced the ELT Merge Into and ELT Insert-Select Snaps to support up to one output view, and added the Get Preview Data check box to these Snaps. You can now connect downstream ELT Snaps to these Snaps.
Updated eXtreme Snap Packs
- Introduced the SCD2 - Spark SQL 2.x Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases in the eXtreme mode.
- Upgraded the /wiki/spaces/AP/pages/410189849to support Spark 3.0.1 on the following cloud platform versions. Ensure that you update your Snaplex to 4.25 GA before running your existing eXtreme Pipelines.
- Amazon EMR 6.2.0 (Hadoop distribution: Amazon)
- Azure Databricks 7.5
Known Issues
- When ELT and Spark SQL 2.x Snap account credentials—such as user names, passwords, client secrets, auth codes and tokens, secret keys, and keystores—are auto-filled using the Google Chrome browser, the accounts, and hence the Pipelines fail. This is because the browser overwrites the field values with its own encrypted values that the SnapLogic Platform cannot read. SnapLogic recommends that you do not auto-save your Snap account credentials in the Chrome browser.
Ensure that you delete any credentials that the browser has already saved for elastic.snaplogic.com, and then perform ONE of the following actions:- Option 1: Click