May 2021, 4.25 Release Notes

In this article

Key New Features in SnapLogic

Platform:

  • Support for elastic scaling in Kubernetes
  • Enhanced memory and resource management
  • Anomaly detection and active Pipeline analysis
  • Additional notifications via Slack

API Management:

  • Support for creating APIs from Swagger 2.0 specifications

  • Support for API versioning 

ELT for Cloud Data Platforms:

  • New! Launching Fast Data Loader
  • Introducing two new Snaps -- ELT Execute and ELT SCD2
  • Support for Snowflake that is hosted on Azure or Google Cloud Platform (GCP)

  • Support for ELT operations on data from Google Cloud Storage into Snowflake

  • Automatic download of JDBC Drivers for Snowflake, Redshift, and Azure Synapse 
  • Support for additional Window functions: Covariance, Correlation, and Linear Regression Functions

New Snaps:

  • Twilio Snap Pack

  • NetSuite Call RESTlet Snap
  • SAP RFC Listener Snap


Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the main-9727 - Patch 1 version at 9 p.m. PT on June 12, 2021. Orgs migrated to the 4.25 GA version prior to the auto-upgrade are not impacted. 
  • To use the new SnapLogic features and Snaps in the 4.25 release, ensure that your Snaplex nodes are upgraded to the recommended version.
  • This 4.25 release includes the patches deployed to SnapLogic after the 4.24 release, as listed in the 4.24 Dot Releases section. 
  • We remind you to add the IP addresses given in the SnapLogic allowlist article to your network.


4.25 Dot Releases

Platform

Date of UpdateSnaplex BuildUpdates
main-9988 - 4.25 Patch 3 
(Recommended)
  • Fixed an issue where Tasks with the + symbol in their names did not complete execution when activated.
  • Fixed an issue where service directory errors caused log files to increase in size.
  • Fixed an issue where memory usage for a Snaplex node increased despite a lack of Pipeline activity, unless restarted.
  • Fixed an issue where some Groundplex nodes experienced unexpectedly high CPU utilization.
  • Fixed an issue where Scheduled Tasks were not scheduled correctly because of daylight savings time.
  • Fixed an issue where some OAuth 2.0 endpoints did not support additional properties sent in the client body with credentials.
N/A

Control Plane update:

  • Fixed an issue where the public API for Pipeline runtime collection returned less runtimes than expected.
  • Fixed an issue where Task notifications were not taking effect despite being created successfully.
  • Fixed an issue in Designer where suggested Snaps for ELT Pipelines included standard-mode Snaps, even though they are not supported.
 main-9825 - 4.25 Patch 2 (Restricted)This is a restricted Snaplex build. For more information, contact support@snaplogic.com.
 main-9727 - Patch 1 
  • Fixed an issue where some recurring Scheduled Tasks caused duplicate Pipeline executions.
  • Fixed an issue where the Do not start a new execution if one is already active option in Scheduled Tasks was not enabled.
  • Fixed an issue with the OAuth 2.0 implementation for some Snap accounts by adding support to send client credentials as a Basic Auth header.
  • Fixed an issue with Snaps that allocate Snap metrics, where over time a memory leak occurred, requiring the reboot of the Snaplex node associated with the Pipeline.
 N/A

Control Plane update:

  • Fixed a caching issue with the UI.

  • Fixed an issue with the Project Migration API by restoring the original functionality of the duplicate_check parameter.
  • Fixed an issue where after overwriting a file in SnapLogic Manager successfully, the changes were not reflected in the actual file and the data remains unchanged when the file was reopened.

UI

Date of UpdateUpdate
 
  • Fixed an issue where opening the document preview for the second input of an Open API Snap caused the UI to freeze.
  • Fixed an issue where Pipeline Parameters were not displayed in the dialog window for new Triggered Tasks.
 
  • Fixed an issue with account validation where the Snaplex dropdown list Manager was empty when the Project Space or project name had special characters that required encoding.
  • Fixed an issue on the Create API form where re-uploading an API specification and clicking Cancel causes an unexpected error message.
  • Fixed an issue where the Task dialog window did not display the When Task Has options if you entered recipients in the Slack Notification field but left the Email Recipients field empty.
  • Fixed an issue where the Version Details page in the new API Manager console did not indicate on which server the API version was deployed.
  • Fixed an issue with the Dashboard > Snaplex tab where after selecting a Snaplex, all graphs displayed data for a week instead of the default time period (one hour).
  • Fixed an issue where the Triggered Task Details page did not display the Download the CA certificate option.
  • Fixed an issue where a Snap in the New Form UI did not display the default value for a new field in the Settings dialog window.
  • Fixed an issue where settings and recipients entered in the Task execution duration - time limit notification would not be recorded after reopening the completed notification.
  • Fixed an issue where the Mapper Snap Settings dialog window displayed the Missing property value error message twice after you deleted an existing expressions field.
  • Fixed an issue where the red arrow error indication was displayed in the password complexity tooltip.f
  • Fixed an issue with eXtreme Pipelines where the Disable Snap option (which did not function) was removed from the Snap Settings dialog window.
  • Fixed an issue where copying and pasting a Snap with a mouse caused an error, even after clearing the browser cache.
  • Fixed an issue where in some account types the Create Account dialog window did not display the username for the account.
  • Fixed an issue with the Select a Snaplex for validation dialog window where the Snaplex dropdown list displayed only Snaplex instances from the global shared folder.

Standard-mode Snaps

Snap PackDate of UpdateSnap Pack Build NameUpdates
Box425patches10840

Modified default values for the following fields in Box Read and Box Write Snaps:

  • Number of Retries - Changed from 3 to 0.
  • Retry Interval (seconds) -  Changed from 5 to 1.

For existing Pipelines which have the Enable Staging (deprecated field) checkbox selected, this Snap retries the connection once even if the Number of Retries is zero.

DynamoDB 
Flow
JIRA 
Microsoft OneDrive
Teams
SAP
 425patches10571
  • Updated the AWS SDK from version 1.11.688 to 1.11.1010 in the DynamoDB Snap Pack and added a custom SnapLogic User Agent header value.
  • Enhanced the performance of Binary Router Snap through an efficient IO utilization.
  • Enhanced the JIRA Snap Pack to support HTTP proxy authentication.
  • Updated the error output view of the OneDrive - Upload File Snap from a Binary view to a Document view.
  • Improved the error messages in the Teams - Create Team Snap where the Snap fails with a null pointer exception error when the given account information is invalid.
  • Renamed the RFC Listener Snap to SAP RFC Listener Snap. 
Amazon SQS
Binary 
Coupa 
Exchange Online
ML Data Preparation
Slack 
SOAP
Tableau
Workday Prism Analytics 
 425patches10994
  • Enhanced the AWS S3 and Amazon SQS accounts to add a Snap’s information to the User Agent header for Cross IAM account requests.
  • Removed duplicate strings in a request’s User Agent header sent by Amazon SQS or Binary S3 Snaps.
  • In the Binary Snap Pack:
    • Enhanced the Directory Browser Snap to process files with the owner field for the SFTP protocol. Upon validation, the output of the Snap fetches the owner’s name, which was previously displayed as unknown.
    • Fixed an issue with the File Writer Snap where the Snap fails to connect to the SFTP endpoint using the AWS Transfer Family.
    • Fixed an issue where Pipelines fail to connect the SFTP Server with the Binary Snap Pack after upgrading to the SnapLogic 4.25 release version (main9554).
  • Enhanced the Coupa Snap Pack to support HTTP proxy authentication.
  • Improved the Exchange Online Snap Pack to handle authentication token errors and acquire new tokens and improved the error messages in the Create Event Snap where the Snap fails with a null pointer exception error when the given account information is invalid.
  • Fixed an issue with the Deduplicate Snap where the Snap breaks when running on a locale that does not format decimals with Period (.) character. 
  • Improved the error messages in the Channel Operations Snap where the Snap fails with a null pointer exception error when the given account information is invalid.
  • Enhanced the SOAP Snap Pack to support HTTP proxy authentication.
  • Fixed an issue with Tableau REST Snap where the Snap was not able to filter or route the output data shows either duplicate records or incorrect results.
  • Enhanced the Workday Prism Analytics Bulk Load Snap to support HTTP proxy authentication.
JDBC 
MySQL
Oracle
Redshift
Teradata
 425patches11008
  • In the JDBC Snap Pack:
    • Fixed an issue with the Generic JDBC - Select Snap where if there are URL properties in the account, the unPooledConnection does not use the properties when creating a connection. This causes the auto-detect functionality to use the wrong database name.
    • Fixed an issue with MySQL and MariaDB in the Generic JDBC - Update Snap when a table is incorrectly reported as not being found.
  • In the MySQL Snap Pack:
    • Enhanced the MySQL - Lookup Snap suggestions to load the Output Fields and Lookup Column settings.
    • Fixed an issue with the MySQL - Select Snap where the second output view of the Snap was showing empty column values and improved error messages in the Snap where the Snap fails with a null pointer exception error when the given account information is invalid.
  • In the Oracle Snap Pack:
    • Improved the support for the CLOB data type in the Oracle - Insert Snap. 
    • Improved the error messages for all the Snaps in the Oracle Snap Pack where the Snaps fail with a null pointer exception error when the given account information is invalid. 
    • Updated documentation links for Oracle Thin Account and Oracle Thin Dynamic Account in the Snap settings. 
  • Updated the AWS SDK from version 1.11.688 to 1.11.1010 in the Redshift Snap Pack and added a custom SnapLogic User Agent header value.
  • Improved the error messages for all the Snaps in the Teradata Snap Pack where the Snaps fail with a null pointer exception error when the given account information is invalid. 
Microsoft Dynamics CRM  425patches10880Improved the Dynamics 365 for Sales Upsert Snap by caching the parameter lookup values.
Azure Active Directory 425patches10868Enhanced the Create Entry and Update Entry Snaps to support Pipeline parameters and upstream values for the Attribute name setting under Attributes and the Snap Pack to support proxy authentication. 
SQL Server 425patches10797

Fixed an issue with the SQL Server - Bulk Load Snap where the Snap fails when the login password contains a colon or a less than (<) symbol.

REST

425patches10419

Fixed a memory leak issue in the REST Snap Pack.
Anaplan 425patches10360

Added the following fields to the Anaplan Write Snap:

  • Override server settings
  • Separator
  • Text Delimiter
Salesforce 425patches10182Fixed an issue with the Salesforce Read Snap where the retry fields do not work as expected when a Pipeline has more than ten Snaps and improved the error messages in the Snap where it fails with a null pointer exception error when the account reference provided is invalid.
REST 425patches9929Enhanced the REST Snap Pack to support Proxy Authentication.
Transform 425patches10663

Fixed an issue in the CSV Formatter Snap where even if the Ignore empty stream checkbox is not enabled, the Snap did not produce an empty binary stream output in case there is no input document.

Reltio 425patches10410Fixed an issue with the Reltio Create Snap where the Snap processes a lesser number of documents while loading data into Reltio.
Transform 425patches10152

Enhanced the CSV Parser Snap with a new checkbox Preserve Surrounding Spaces that enables you to preserve the surrounding spaces for the values that are non-quoted.

Replaced the Strict XSD output field with Map input to first repeating element in XSD field in the XML Formatter Snap. If selected, the Snap ignores the root element from the XSD file.

SOAP 425patches10152

Enhanced the SOAP Execute Snap with a new checkbox Escape special characters that enable the snap to escape XML special characters in XML template variable values when this Snap inserts these values into the Apache Velocity template.

Snowflake 425patches10190Enhanced the Snowflake S3 Database and Snowflake S3 Dynamic accounts with a new field S3 AWS Token that allows you to connect to private and protected Amazon S3 buckets.
Confluent Kafka 425patches10543
  • Improved the handling of interrupted/aborted Kafka Snaps to ensure proper clean-up of metrics.

  • Optimized the Kafka Producer Snap to initialize the Kafka API only if there is at least one input document.

  • Fixed an issue of account passwords being included in the log messages output of Kafka Snaps. The password is now hidden in the logs for both Kafka Consumer and Kafka Producer Snaps.

  • Enhanced the Kafka Producer Snap to support Avro serialization format with a new target data type Avro under Key and Value Serializer.
  • Enhanced the Kafka SSL Account with new fields (Registry Username or Key and Registry Password or Secret) for Schema Registry authentication. The two existing SASL properties (SASL Username and SASL Password) have been revised to SASL Username or Key and SASL Password or Secret, respectively.

  • The Kafka Producer and Consumer Snaps create a default client ID string to identify the specific Snap instance.
  • Apache Kafka client library is upgraded from version 2.6.0 to 2.8.0.

  • Confluent Kafka client libraries are upgraded from version 5.2.1 to 6.2.0.

OpenAPI 425patches10417

Enhanced the OpenAPI Snap with the following two fields:

  • HTTP Headers
  • Query Parameters
Salesforce 425patches9609Enhanced the Salesforce Read Snap to allow you to add an optional second output view that exposes the schema of the target object as the output document. The output view of the Snap has a minimum of one output and a maximum of two outputs. 
Microsoft Dynamics CRM 425patches9813

Fixed an issue with DEPRECATED | Dynamics CRM Search Snap where the Snap returns duplicate records when reading results from multiple pages.

Binary  425patches10217
  • Fixed the following issues in the PGP Decrypt Snap caused due to the Snap patch 425patches9581:
    • The Snap writes an empty document to the output view when there is an error in the Snap. It will now decrypt the document and write it to the output view only if it succeeds.
    • The Snap fails to decrypt any non-signed encrypted file more than 500 bytes. It will now decrypt both non-signed and signed encrypted files. 
  • Removed the Project ID field from the Google Service Account in the Binary Snap Pack, since the JSON Key already contains information about the Project ID. 
MS Dynamics 365 for Sales 425patches10303

Fixed an error in the Dynamics 365 for Sales Create Snap where the Snap does not send the original documents to the error view in the output when the batch size property is set to greater than 1.

ServiceNow 425patches10014

Enhanced the ServiceNow Query Snap with the retry mechanism that includes the following settings:

  • Number of retries: Specifies the number of attempts the Snap should make to perform the selected operation in case of a connection failure or a timeout.
  • Retry interval (seconds)Specifies the minimum number of seconds the Snap must wait before each retry attempt.
Box  425patches10072Modified default values of the following fields across the Box Snap Pack to trigger automatic retries by default, in case of connection failures or timeouts:
  • Number of retries – Changed from 0 to 3
  • Retry interval (seconds) – Changed from 1 to 5
Hadoop 425patches9975

Fixed a dependency issue in Hadoop Parquet Reader Snap while reading data from the AWS S3 folder. The issue is caused due to conflicting definitions for some of the AWS class dependencies in the classpath.

PostgreSQL 425patches9879

Enhanced the performance of PostgreSQL - Bulk Load Snap significantly. We expect the Snap to execute up to 3 times faster than the previous version for enterprise workloads.

Transform 425patches9815Fixed a ClassCastException error in the Avro Parser Snap and handling of the map, fixed, enum, and bytes data types in the Avro Formatter and Avro Parser Snaps.
Binary 425patches9581

Fixed an issue with the PGP Decrypt Snap where the Snap writes an empty document to the output view when there is an error in the Snap. It will now decrypt the document and write it to the output view only if it succeeds.

As of  we have discovered an issue with this Binary patch affecting the PGP Decrypt Snap. In case your Pipelines use the PGP Decrypt Snap, we recommend you to NOT use this patch. Else, your Pipelines will likely fail. We're working on a fix and will be deploying a Snap patch soon.

Transform 425patches9749Enhanced the XML Parser Snap to recognize input headers when defining the inbound schema.
Amazon SQS 425patches9596Added the Security token field to the Amazon SQS Account.
REST 425patches9684
  • Fixed the REST Post Snap that fails to correctly set the Content-Length header when working with multi-file uploads.
  • Added Send Client Data as Basic Auth Header check box to the Create Account settings of the OAuth2 and OAuth2 SSL accounts. This check box enables you to include and pass the client credentials information in the header when connecting to the OAuth endpoint.
  • Enhanced the REST Snap Pack with the new account REST AWS Sig V4 Account that adds the ability to call any AWS API.
Binary 425patches9564Enhanced the SFTP session in Binary Snaps to support the SFTP servers that close the SFTP sessions slowly.
Transform 425patches9638

Reverted the Join Snap to the 4.24 release behavior. This is in response to an issue encountered in the Join Snap in the 4.25 release version (main9554) that can result in incorrect outputs from all Join Types. 425patches9638 is the default version for both stable and latest Transform Snap Pack versions for orgs that are on the 4.25 release version. No action is required by customers to receive this update and no impact is anticipated. For more information, contact support@snaplogic.com.

Data Automation

Snap PackDate of UpdateSnap Pack Build NameUpdates
ELT 425patches10017
  • Updated the ELT SCD2 Snap to replace End date of historical row option in the Meaning field of Target Table SCD2 Fields field set with End Date of Current Row.

    Breaking change

    This may cause the existing Pipelines to fail as the End date of historical row option no longer exists.

    You need to make the following update in the ELT SCD2 Snap's settings across your Pipelines after upgrading your Snap Pack to this patch version:

    • Select End Date of Current Row from the Meaning drop-down list in the second entry (highlighted in the image).
  • Fixed the issue with the ELT Insert-Select Snap containing an open output preview that fails to retrieve output preview data in case of Redshift and Azure Synapse databases, though the Pipeline runs work as expected.
  • Fixed an issue where the ELT Execute Snap does not error out (Snap turns Green) even when running an SQL query to drop a non-existent table from a Snowflake or Azure Synapse database.

  • [Update on ]: Enhanced the ELT Snap previews to support the following data types: array, object, variant, and timestamp.

    • The Snaps convert the values to hexadecimal (HEX) equivalents—the default setting for the session parameter BINARY_OUTPUT_FORMAT in Snowflake. See Session Parameters for Binary Values for more information.

    • If this setting is different from hexadecimal (such as base64) in the Snowflake table, the Snaps still convert the values to hexadecimal equivalents for rendering them in the Snap previews.

ELT
 
425patches9725
  • Enhanced the ELT Snap preview to display the exact binary and varbinary values from the Snowflake database during Pipeline validation, by converting the values to hexadecimal equivalents—the default setting in SnowflakeIf the setting is different from hexadecimal in the Snowflake table, then the Snaps still convert the values to hexadecimal for rendering the Snap preview.
  • Enhanced the ELT Transform Snap to display the appropriate data type (binary or varbinary) for the column names populated in the output schema.
  • Enhanced the ELT Window Functions Snap to address potential issues due to an incorrect definition for MINUS function in the case of Redshift and Azure Synapse databases.


4.25 UAT Delta

 Click here to expand...

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT 1: Friday, April 23, 2021 – Snaplex version: 4.25_rc-9551 - 4.25 GA (Recommended)

  • UAT 2: Friday, April 30, 2021 
  • GA: Saturday, May 08, 2021 (9 p.m. PT)  

The SnapLogic GA release build on the production server is also deployed on the UAT server.

UAT 2 Fixes

Platform

Design-first API Management:

  • Fixed an issue with the API specification that fails to import while updating a Pipeline.
  • Fixed an issue where a user with full access to the SnapLogic Platform is unable to create an API.

UI

  • Fixed an issue with the new Task dialog window where you cannot assign another Snaplex to the task when the original Snaplex is deleted.
  • Fixed an issue with Scheduled Tasks in Manager that opens the Task Update dialog in the old UI, instead of the new UI.

Snaps


Platform 

Snaplex-based Scheduler

The Snaplex-based Scheduler will be the default scheduling mechanism for all Scheduled Tasks in the 4.26 release (August 2021). The new scheduling mechanism improves the timeliness and reliability of Scheduled Task executions, and we recommend that you enable the Snaplex-based Scheduler on your Orgs.

New Features

  • API Management - Design-First: Introduced an API Management section in Manager where you can create an API based on an Open API Specification (OAS) 2.0. As an Org admin, you can also manage APIs in the new API Manager console. The API Manager - /apim space supports the following activities:

  • Support for elastic scaling in Kubernetes: Introduced autoscaling for your Snaplex instances that are running in a Kubernetes environment. This implementation replaces the previous Kubernetes-based deployment of a Snaplex. Starting with the 4.25 release (May 2021), you can set up your Snaplex helm chart to initiate the scaling up and down of your node resources through Kubernetes orchestration. This implementation requires the Prometheus adapter and Grafana analytics.

  • Anomaly detection and active Pipeline analysis: Introduced two alert notifications for tracking Task executions that take a longer time than usual. You can define a hard limit either in minutes or by the percentage based on the last 30-day historical executions.


  • Enhanced memory and resource management to limit memory growth on Snaplex nodes. If the memory usage exceeds 95% of the configured maximum heap limit, the Pipelines that are causing the high memory utilization are terminated to prevent the Snaplex node from failing with an OutOfMemory error.

Enhancements to Existing Features

  • Snaplex monitoringAdded support for granularity in the reporting of data spikes in your Pipeline executions in the widgets on the Snaplex Dashboard Wall. You can now select Pipelines within a 15-minute window (indicated by the vertical line) and correlate active Pipelines with high resource utilization.


  • Additional notifications via Slack: Enhanced support for in-platform notifications. You can now add Slack recipients (channels and direct users) to Org-wide Snaplex notifications and Task notifications.
     
  • Asset Palette for APIs: Support for searching for Pipelines that are used by APIs that are created in the new API Management console.


  • Progressive loading of the Asset tree in Manager: Enhanced the search-and-browse experience in Manager. The assets are retrieved as needed, thus reducing latency.

  • UI refresh of Task dialog windows: Redesigned dialog windows for Triggered, Ultra, and Scheduled Tasks, giving the displays the same appearance as that of the New Form UI. As an enhancement, you can now select the Task type when opening the dialog window from Designer.

  • UI refresh of the Account dialog windows: Updated the UI and enhanced the support for creating accounts in Manager. Previously, in the 4.24 release and earlier, the option to create or modify an account using the New Form UI was available only in Designer.

Known Issues

  • When you migrate a project more than once, the existing assets are overwritten. You might have to associate your Tasks with the correct Pipelines after the migration. Typically, even after multiple migrations, account references remain intact in the Pipeline, but we recommend that you verify all references before starting your Pipeline executions. This behavior is a result of the duplicate_flag parameter being disabled.

  • When a start time of a Scheduled Task using the Snaplex-based scheduler is modified, the configured start time is not used to trigger the Pipeline execution.

  • In some scenarios with the Snaplex-based Scheduler, multiple executions of a Scheduled Task occur when only one instance is set to execute.

  • When calling a child Pipeline from a Groundplex to Cloudplex or vice versa, the Pipeline Execute Snap hangs, causing the parent Pipeline not to complete the execution.

  • The New Form UI does not show the default value for a custom property when editing an existing instance of a Snap.

  • In Manager, the Task Details page for some Triggered Tasks does not display the Download CA Certificate option.

  • When creating iterative Scheduled Tasks in time zone regions that have Daylight Savings, we recommend that you select the Cron Job option. See best practices for details.

Behavior Change

Retired the legacy UI: The legacy UI for Snaps is retired. Starting in the 4.25 release (May 2021), the option to deselect the New Form UI in the User Settings dialog window is no longer available. All Snaps can be viewed only in the new UI.


Standard-mode Snaps

New Snaps

  • NetSuite Call RESTletThis Snap provides the functionality of the NetSuite RESTlet framework that supports the HTTP methods (GET, DELETE, POST, PUT) and MIME types. You can call RESTlet scripts through token-based authentication.
  • SAP RFC Listener: This Snap facilitates Remote Function Calls (RFCs) between two SAP systems. The Snap simulates an RFC server and listens to the incoming RFC calls infinitely. Subsequently, the Snap generates output documents from the import parameters or triggers a child Pipeline. Business applications can use the RFC mechanism to communicate and exchange information (in predefined formats) with other systems.
  • Twilio Snap Pack: This Snap Pack enables you to perform various communication functions programmatically, such as making phone calls and sending text messages using Twilio's Web Service APIs. This Snap Pack has the following Snaps:

Updated Snap Packs

  • Binary Snap Pack:
    • Enhanced the Binary Snap Pack by adding an account Two-Factor Auth Account that supports two layers of authentication for Secure File Transfer Protocol (SFTP). The two secure layers of verification to grant access include the Basic Auth (username/password) and the SSH Key (username/private key) methods.
    • Enhanced the S3 File Writer Snap with the following abilities:
      • Use an MD5 checksum that checks for data integrity and corruption while uploading the file.
      • Write S3 objects with Access Control List (ACL) details in a single API call.

        Behavior Change

        In the 4.23 release, the S3 File Writer Snap used to write an S3 file and then set ACL if the ACL property was not empty. If the file write was successful and the ACL setting failed, both the output document in the output view and the error document in the error view was displayed. This issue has been resolved in the 4.24 release by enabling the Snap to configure ACL while writing the fileTherefore, the Snap produces an error document in the error view if the ACL configuration fails, irrespective of whether the file write action passes or fails.

  • Box Snap Pack:
    • Enhanced the Snaps in this Snap Pack with the Retry mechanism that includes the following settings:
      • Number of retries: Specifies the number of attempts the snap should make to perform the selected operation in case of connection failure or timeout.
      • Retry interval (seconds)Specifies the minimum number of seconds the Snap must wait before each retry attempt.
    • Enhanced the output view of the Box Directory Browser Snap to display the error message, Error retrieving contents of the folder with id:<Folder ID> instead of No folder exists with <Folder ID> when you enter an invalid folder ID.

  • DynamoDB Snap PackEnhanced the DynamoDB Account to support the Expression Enabler for account properties and creation of dynamic accounts.

  • Flow Snap PackEnhanced the HTTP Router Snap to support Ultra Pipelines.

  • Google Directory Snap Pack: Enhanced the Google Directory Insert and Google Directory Update Snaps to support custom schema attributes. You can fetch these custom fields and their values in a user profile by adding them as custom schema attributes to upstream Snaps.

  • Hadoop Snap Pack: Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
    • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
    • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.

  • Microsoft OneDrive Snap PackEnhanced the Update Share Permissions Snap with the Share Type and Expires fields that enable you to update the share permissions on your OneDrive items shared using the Invite or Link option. If the Share type is link, you can specify the expiry date and time for the URL that is used to share permissions.

  • Microsoft Power BI Snap Pack: Enhanced the following Snaps in the Microsoft Power BI Snap Pack by supporting data flows with two options, Dataflow and Dataflow Storage Accounts.
  • Salesforce Snap Pack:

    • Enhanced the pass-through behavior in Bulk API mode for Salesforce Write Snaps to include the input document in the original key when the batch size is set to 1. When the batch size is greater than 1, the input document is not included in the original key. This enhancement impacts the following Snaps:

    • Enhanced the Salesforce Upsert Snap to support CSV payload using the new field Bulk Content Type in Bulk API mode. The Salesforce API has recently introduced support for CSV payloads. In the 4.24 release and earlier, this Snap supported only the XML data type; starting from the 4.25 release, you can select CSV as the Bulk Content Type drop-down list. The default content type is set to XML to support backward compatibility. This enhancement also addresses the null values issue in Bulk API mode.
    • Enhanced the Snaps in the Salesforce Snap Pack by updating the field labels to the title case. If you have existing Pipelines that use Salesforce Snaps that include the field names in error messages, you must update those Pipelines to use the new field labels.

  • Snowflake Snap Pack:

    • Enhanced the Snowflake - Bulk Load Snap to enable data transformation using the new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. It also enables you to transform data types. This Snap supports CSV and JSON file formats for data transformation.

  • Transform Snap PackEnhanced the Group By N Snap with the following settings: 

    • Memory Sensitivity: The Snap's response to the memory changes. 
    • Group Size: The maximum number of input documents to be grouped into a single output document.
    • Min Group Size: The minimum number of input documents to be grouped into a single output document.
    • Flush Timeout: The number of seconds the Snap must wait before processing the records even if the incoming messages do not reach the specified Group Size.

Known Issues

Prior 4.24 Snap Dot Releases that are Merged into 4.25 GA

 Click here to expand...

Snap PackDate of UpdateSnap Pack Build NameUpdates
Binary 424patches9044Enhanced the S3 File Writer to use an MD5 checksum that automatically checks for data integrity and corruption while uploading the file.  
Box 424patches9242
  • Enhanced the Box Snap Pack with the Retry mechanism that includes the following settings:

    • Number of retries: Specifies the number of attempts the snap should make to perform the selected operation in case of connection failure or timeout.
    • Retry interval (seconds)Specifies the minimum number of seconds the Snap must wait before each retry attempt.
Hadoop 424patches9262

Enhanced the AWS S3 Account for Hadoop to provide role-based access when you select the IAM role check box.

Anaplan 424patches9313Fixes an issue related to Index out of range in the Anaplan Upload Snap when the separator is not present or does not match the one in the upload file. 
Binary 424patches9020

Fixes the file operation error of copying large S3 files in the File Operation Snap by supporting the transfer of zero-byte S3 files while uploading the file.

Snowflake424patches8905

Enhances the Snowflake - Bulk Load Snap to allow transforming data using a new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. This Snap supports CSV and JSON file formats for this data transformation.

DynamoDB 424patches9031
  • Supports autoscaling by adding retry logic with exponential backoff to the Snaps.
  • Enhanced the DynamoDB Account to support the Expression Enabler for account properties and creation of dynamic accounts.
  • Improves memory consumption in DynamoDB Bulk Write and Bulk Get Snaps.
SOAP  424patches9077
  • Fixes the incremental thread count issue in the SOAP Execute Snap by ensuring that only instance of SoapCustomAsyncHTTPConduitFactory should pass throughout the execution of the Snap.
  • Fixes an issue in SOAP Execute Snap where the Snap fails as the default value of Content-Type field was modified after upgrading the Snap in the 4.24 release.
JMS 424patches9098Enhances the JMS account in the JMS Snap Pack by updating the JNDI properties (java.naming.security.principal, java.naming.security.credentials) with username and password, to support the requirements of certain JMS servers.
JMS 424patches9104Fixes an issue in the JMS Consumer Snap, where different JMS SQS accounts interfere with each other when running on the same JCC node by using the AWSStaticCredentialProvider instead of the SystemPropertiesCredentialsProvider.
Salesforce  424patches9024
Microsoft Power BImain80

Enhances the following Snaps in the Microsoft Power BI Snap Pack by adding two values, Dataflow and Dataflow Storage Accounts to the Entity Type field.

Twilio main80Introduces the Twilio Snap Pack that enables you to perform various communication functions programmatically, such as making phone calls and sending text messages using Twilio's Web Service APIs. This Snap Pack offers the following Snaps:
SQL Server 424patches8657

Enhances the SQL Server - Select Snap by introducing a new field, Query Hints, which helps to customize and optimize the database engine to process query statements.

Exchange Online 

 

424patches8965

Fixes an issue in the Snap Pack by handling the conflict of users having same email IDs by accepting only the first user’s email ID returned by the API and ignoring the duplicate ones.

/wiki/spaces/SD/pages/1438719 424patches8617Fixes an error handling issue in the PostTweet Snap, where data and error are now routed to the error view.
Transform424patches8938
  • Fixes the timestamp issue in JSON Formatter Snap that changed the time zone offset to include colon by default after upgrading to 4.24.
  • Fixes the null pointer exception at runtime in Fixed Width Parser Snap by setting the Trim column data field to false for empty columns.
  • Enhances the Group By N Snap to process the records efficiently by adding a Flush Timeout field that enables the Snap to flush a partial group of records if the time specified in this field passes with no new input.
NetSuite424patches8891

Introduces the new Snap NetSuite Call RESTlet that enables you to call RESTlet scripts through token-based authentication. This Snap provides the functionality of the NetSuite RESTlet framework that supports the HTTP methods (GET, DELETE, POST, PUT) and MIME types.

RabbitMQ424patches8750

Fixes the handling of RabbitMQ Headers in the RabbitMQ Consumer Snap, including the x-death header.

JDBC424patches8903Fixes the Pipeline execution failure issue when using the JDBC Snaps with a Microsoft Access account by setting the correct quotation handler for the configured account.
Binary

Hadoop

Hive


424patches8876
  • Fixes the missing library error in Binary, Hadoop, and Hive Snap Packs when running Hadoop Pipelines in JDK11 runtime.
  • Enhances the Binary Snap Pack by adding a new account - Two-Factor Auth Account that supports two levels of authentication (2FA) for SFTP protocol to access SFTP.
Salesforce 424patches8569

Fixes an issue in Salesforce Bulk Upsert, Salesforce Bulk Update, and Salesforce Bulk Create Snaps that fail to process Related object and Related external ID values by modifying the input schema that formats the CSV data.

Potential Breaking Change (now fixed in 424patches9024)

This Salesforce Snap patch contains an issue affecting the Salesforce Subscriber and Salesforce Publisher Snaps. If you do not use these Snaps in your Pipelines, then you can use this patch version. Else, we recommend you to not use this patch version as Pipelines will fail. 

Confluent Kafka 424patches8805Fixes an issue with the Kafka Producer Snap by removing the account validation, which is not required, when the Snap initializes in a Pipeline execution.
Teradata 424patches8799
  • Enhances the Teradata Execute Snap by adding a field, Advanced options, which extends support with microsecond precision for TIMESTAMP data type.
  • Fixes an issue in the TPT Load Snap where the Snap fails to load data into the table when creating an output.
  • Fixes an issue with the TPT Load Snap where both null and empty string values loaded as null, such that a null value loads as null and an empty string loads as empty string, respectively.

    • This fix may cause existing Pipelines to fail if empty string values are expected to be loaded as null.

    • Following are the new configurations:

      • VARCHAR QuotedData= Optional

      • VARCHAR OpenQuoteMark= \”

      • VARCHAR NullColumns= Yes

Google Directory 
 
424patches8638

Enhances the Google Directory Read Snap by introducing two field attributes, projection and customFieldMask, to retrieve custom schema fields.

Data Automation

SnapLogic's data automation solution speeds up the identification and integration of new data sources, and the migration of data from legacy systems. The solution can automatically detect duplicate, erroneous, or missing data, and identify structures and formats that do not match the data model. Data automation can accelerate the loading and transformation of your data into the data warehouse, speeding up the data-to-decisions process.

New Features in ELT Snap Pack

  • Starting with the 4.25 release, SnapLogic has now certified the ELT Snap Pack to work with Snowflake hosted on Google Cloud Platform (GCP) as the target database, in addition to the other flavors of Snowflake hosted on AWS and Microsoft Azure
  • Introduced the ELT Execute (Archived) Snap to enable you to run DML, DDL, and DCL SQL queries in Snowflake in Snowflake, Redshift, and Azure Synapse.
  • Introduced the ELT SCD2 Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases—Snowflake, Redshift, and Azure Synapse.
  • Enhanced the ELT Database Account to introduce:
    • Support for Google Cloud Storage as a storage location (source) in addition to AWS S3 and Azure Data Lake Storage (ADLS) when your target database is Snowflake.
    • Automatic download of the JDBC driver required for the selected Database Type using the new Download JDBC Driver Automatically check box.

Supported JDBC JAR Versions

ELT Snaps automatically use a corresponding JDBC JAR file to connect to your target database and to perform the load and transform operations

DatabaseCertified JDBC JAR File
Azure Synapsemssql-jdbc-9.2.1.jre8.jar
Redshiftredshift-jdbc42-2.0.0.2.jar
Snowflakesnowflake-jdbc-3.13.1.jar

Using Alternate JDBC JAR File Versions

Though we recommend you to use the above JAR file versions, you can choose to use a different version, based on your environment. 

Updated ELT Snaps

  • Enhanced the ELT Load Snap to prevent changes to existing tables during Pipeline validation. If you set the Load Action as Drop and Create table, and the target table does not exist, the Snap creates a new (empty) target table based on the schema specified in its settings.
  • Enhanced the ELT Window Functions Snap to support Covariance, Correlation, and Linear Regression Functions on Snowflake, Redshift, and Azure Synapse databases. The Snap uses function-specific query re-writes to support these functions on Redshift and Azure Synapse databases.
  • Enhanced the ELT Merge Into and ELT Insert-Select Snaps to support up to one output view, and added the Get Preview Data check box to these Snaps. You can now connect downstream ELT Snaps to these Snaps.

Updated eXtreme Snap Packs

  • Introduced the SCD2 - Spark SQL 2.x Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases in the eXtreme mode.
  • Upgraded the /wiki/spaces/SD/pages/410189849to support Spark 3.0.1 on the following cloud platform versionsEnsure that you update your Snaplex to 4.25 GA before running your existing eXtreme Pipelines.
    • Amazon EMR 6.2.0 (Hadoop distribution: Amazon)
    • Azure Databricks 7.5

Known Issues

  • When ELT and Spark SQL 2.x Snap account credentials—such as user names, passwords, client secrets, auth codes and tokens, secret keys, and keystores—are auto-filled using the Google Chrome browser, the accounts, and hence the Pipelines fail. This is because the browser overwrites the field values with its own encrypted values that the SnapLogic Platform cannot read. SnapLogic recommends that you do not auto-save your Snap account credentials in the Chrome browser.
    Ensure that you delete any credentials that the browser has already saved for elastic.snaplogic.comand then perform ONE of the following actions:
    • Option 1: Click  that appears in the address bar after you submit your login credentials at elastic.snaplogic.comand then click Never
    • Option 2: Disable the Offer to save Passwords option at chrome://settings/passwords while working with your SnapLogic Pipelines. If you disable this option, your Chrome browser will not remember your passwords on any other website.
  • For Redshift and Azure Synapse databases, the ELT Load Snap displays JSON values for fields with datatypes Date and Time when validating the Pipeline.

    SnowflakeRedshiftAzure Synapse
    ”time_data” : “09:23:56”,”time_data” : {“time”: “09:23:56”},”time_data” : {“time”: “09:23:56”},
    ”date_data” : “2020-12-20”,”date_data” : “2020-12-20”,”date_data” : {“date”: “2020-12-20”},

    However, the ELT Load Snap outputs the correct date and time values when executing the Pipeline. 

    To prevent these values from showing up in JSON format in the previews and ELT Transform Snap, you can cast the time or date values to a VARCHAR type in your custom query as follows: 
    CAST (<time_column_type_name> AS VARCHAR(<size_based_on_precision_required>)).
    Example: CAST (time_data AS VARCHAR(20)).

  • ELT Transform Snap displays incorrect data types (string instead of the actual data type) for column names populated in its Input schema section.

New! Introducing Fast Data Loader (FDL) 

Ideal for business/data analysts, FDL is a data ingestion tool that simplifies the tasks of replicating and synchronizing data from the following source endpoints:

  • Coupa

  • SQL Server

  • Oracle

  • Salesforce

  • ServiceNow

  • Weather (pre-configured endpoint)

  • Covid (pre-configured endpoint)

    You can integrate your data source with the following destination data warehouses:

  • Redshift
  • SAP DWC
  • Snowflake

To gain your complimentary access to FDL, sign-up on the registration page. For more information, see the FDL user guide.  

Prior 4.24 eXtreme Dot Releases that are Merged into 4.25 GA

 Click here to expand...

Snap PackDate of UpdateSnap Pack Build NameUpdates
ELT
 
424patches8793
  • Fixes the column name collision issue in the Snap's output when two tables being joined have columns with the same/identical names. You can specify the extent of prefix (that is, to prefix all columns, only duplicate columns, or no prefix) using the Resultant Column Names Prefix Type drop-down list. Based on the prefix you choose, a table alias name is prefixed to the identical columns in the output.
  • Fixes the issue of production job failures due to ELT Insert Select Snap after upgrading to 4.24 GA by updating the ELT Transform Snap to continue allowing duplication of fields in the expression list for the Pipeline to complete successfully.

No changes are needed to your existing Pipelines.


Behavior Change

The behavior of ELT Load Snap for Load Action during Pipeline validation across the supported databases is as follows:

  • Append rows to existing table: Does not append the data from the source files into the target table.
  • Overwrite existing table: Does not overwrite the data in the target table.
  • Drop and Create table: Does not drop the target table even if it exists, but the Snap creates a new target table if a table does not exist.
  • Alter table: Does not modify the schema of the target table.
/wiki/spaces/SD/pages/410189849
 
424patches8724

Fixes an issue with eXtremeplex that is unable to read Parquet files written from a Groundplex (and hence displays base64 enabled in output columns upon validation), by changing the data encoding from Base64-encoded to plain text format. This issue does not occur during Pipeline execution.