November 2021, 4.27 Release Notes
In this article
Key New Features in SnapLogic
API Management:
Added Developer Portal Accounts and Subscription Manager.
Added Subscription Key Management.
Added improvements for editing and viewing API Specifications.
Developer Portal:
Added Featured API category to help discover and promote APIs.
Support for creating developer accounts to access the catalog and create subscriptions.
Support for creating Applications and subscribe to APIs.
New Snaps/Accounts:
Added Amazon SNS Snap Pack.
Added OAuth Account for Salesforce Snap Pack.
Support for Mutual Authentication in Salesforce Snaps.
Support for Azure Service Bus and Exact Online (coming soon).
ELT for Cloud Data Platforms:
Support for Google BigQuery as a target database.
Support for MERGE INTO ALL option and automatic source table aliasing in ELT Merge Into Snap.
SnapLogic Flows:
Flows is specifically built for business users to more easily automate business processes. Flows greatly simplify integrations for business users while allowing IT to maintain control of security and governance.
Support for new endpoints: Microsoft Dynamics 365 Sales, Microsoft Teams, Microsoft SharePoint, ServiceNow, and SnapLogic Read/Write to Pipeline.
Support for unified error handling for efficient troubleshooting.
Refreshed homepage with improved performance.
Flows is available to you free with your current SnapLogic subscription, but only for a limited time. To get started, register for Flows.
4.27 UAT Delta
Important
Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the recommended version (main-10919 4.27 Patch 3) at 9 p.m. PT on Saturday, Jan 8, 2022. Orgs migrated to the 4.27 GA version prior to the auto-upgrade are not impacted.
The Snaplex upgrade process also upgrades the Java version to the latest version. After the upgrade, the Java version of your Cloudplex nodes may not be the same as your FeedMaster nodes. However, this does not impact the operations of your SnapLogic instance. We will be fixing this issue in a subsequent release.To use the new SnapLogic features and Snaps in the 4.27 release, ensure that your Snaplex nodes are upgraded to the main-10704 (Recommended) version.
This 4.27 release includes the patches deployed to SnapLogic after the 4.26 release, as listed in the 4.26 Dot Releases section.
4.27 Dot Releases
Platform
Date of Update | Snaplex Build | Update |
---|---|---|
Feb 7, 2022 | main-11121 4.27 Patch 4 | Upgraded to Log4j version 2.17.1. |
Dec 20, 2021 | main-10919 4.27 Patch 3 (Recommended) |
|
Dec 16, 2021 | main-10904 4.27 Patch 2 | Fixed a potential vulnerability issue regarding Log4j. Upgrading to Log4j 2.16.0 fixed this issue by removing support for message lookup patterns and disabling JNDI functionality by default. |
Dec 10, 2021 | main-10894 - 4.27 Patch 1 | Fixed an issue with the Log4j 2 (Java logging library) that affected Java applications. For additional details, see our Community blogpost. |
UI
Date of Update | Update |
---|---|
Dec 9, 2021 |
|
Nov 23, 2021 | Fixed an issue where null or empty parameters were being overwritten by the default Pipeline parameters on the subsequent task update. This could cause a problem if you were updating tasks where you set the parameters to null/blank, but needed to update something else on the task. For the triggered task, the empty values are now retained upon updating any fields. |
Snaps
Snap Pack | Date of Update | Snap Pack Version | Updates |
---|---|---|---|
Feb 4, 2022 | 427patches13147 | Fixed an issue with the Asset Read, Lead Read, and Asset Write Snaps where the original payload was not passed through to the error view. | |
Feb 2, 2022 | 427patches12999 | Fixed an issue where the Bulk Lead Extract and Bulk Lead Upsert Snaps failed when the Snaplexes were proxy-enabled. | |
Jan 27, 2022 | 427patches14079 | Fixed an issue with the Salesforce Subscriber Snap where the Snap was unable to re-subscribe to a topic if it did not receive any message for 72 hours — it stopped and displayed an error message. With this fix, the Snap automatically sets the Replay ID value to -2 when:
| |
Jan 25, 2022 | 427patches13769 | Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM. | |
Jan 21, 2022 | 427patches14283 |
| |
Jan 20, 2022 | 427patches14012 | Fixed an issue in the SQS Consumer Snap where the Snap did not read message attributes when the Client-side buffering checkbox is selected. | |
Jan 13, 2022 | 4.27patch14048 | Introduced the Azure Service Bus Snap Pack— a cloud messaging service that connects applications, devices, and services running in the cloud to any other applications or services. This Snap Pack offers the following Snaps: | |
Jan 12, 2022 | 427patches12966 | Enhanced the Avro Formatter Snap to display meaningful error message while processing invalid and null values from the input. | |
Jan 12, 2022 | 427patches13752 | Upgraded Google BigQuery driver to 1.119.0 version to support time partition intervals by MONTH and YEAR. | |
Jan 12, 2022 | 427patches13992 |
| |
Jan 7, 2022 | 427patches13923 |
| |
Jan 7, 2022 | 427patches13948 | Fixed an issue with the following Snaps, where a deadlock occurred when data is loaded from both the input views. | |
Jan 6, 2022 | 427patches13292 | Fixed an issue with the Profile Snap, where the Snap failed to create a temporary directory as the temp file path exceeded 255 characters. | |
Jan 6, 2022 | 427patches13149 | Fixed an issue with the PostgreSQL - Execute Snap, where the Snap failed when using the Delete query with the RETURNING function. | |
Jan 5, 2022 | 427patches13944 | Fixed an issue with Salesforce accounts, where an account password containing special characters and an empty security token was improperly encoded, due to which the account was not validated. | |
Dec 31, 2021 | 427patches13080 | Fixed an XML parsing issue with empty elements for the NetSuite Snaps. | |
Dec 23, 2021 | 427patches13789 | Fixed the following issues with the Salesforce Snaps:
| |
Dec 20, 2021 | 427patches12999 | Fixed an issue with the Redshift Bulk Load Snap, where the temporary files in S3 were not deleted for aborted or interrupted Pipelines. | |
Dec 17, 2021 | 427patches (main96) | Introduced the Exact Online Snap Pack that enables businesses to automate operations in a single flexible subscription, improve business results, and maximize data security. This Snap Pack offers the following Snaps: | |
Dec 17, 2021 | 427patches (main95) |
| |
Dec 16, 2021 | 427patches13615 | Fixed the table truncate 404 error in the Google BigQuery Load (Streaming) Snap by supporting the retry functionality. The Snap now waits in case of an error and retries before loading the data. | |
Dec 13, 2021 | 427patches13539 |
| |
Dec 7, 2021 | 427patches13429 | Updated the Splunk Basic Auth account with the following:
| |
Dec 6, 2021 | 427patches13004 | Fixed an issue with the SMB Account in Binary Snaps where the Snaps could not re-establish a lost connection to the SMB mount point. | |
Dec 2, 2021 | 427patches13072 | Fixed an issue with the NetSuite Search Snap where the Snap failed when the body is not present in the SOAP request. | |
Dec 1, 2021 | 427patches12750 | Enhanced the REST Put Snap with the Upload Transfer Request Type field to support encoding of chunked data transfer and calculating content length. | |
Nov 29, 2021 | 427patches13160 | Fixed an issue with the Anaplan Snaps, where the log messages were stored in | |
Nov 15, 2021 | main12989 | Fixed an issue with the Group by Fields Snap that caused the Snap to abort with an error when executed with zero input documents. | |
Nov 15, 2021 | main12989 | Fixed an issue with the REST OAuth2 and OpenAPI OAuth2 accounts that caused a new feature, OAuth Password Grant Type, to be excluded from the 4.27 GA version. | |
Nov 20, 2021 | 427patches13030 |
| |
Nov 23, 2021 | 427patches12999 | Enhanced SAP RFC Listener, SAP IDoc Listener and SAP IDoc Document Listener Snaps with a new field set Advanced JCo Server Properties that allows you to configure Java Connector (JCo) server-related attributes. | |
Nov 24, 2021 | 427patches12691 | Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap, where the Snap failed with an exception for big query tables. The | |
Nov 23, 2021 | 427patches12999 | Enhanced the Parquet Reader Snap with int96 As Timestamp checkbox, which when selected enables the Date Time Format field. You can use this field to specify a date-time format of your choice for int96 data-type fields. The int96 As Timestamp checkbox is available only when you deselect Use old data format checkbox. | |
Nov 23, 2021 | 427patches12999 | Enhanced the Snowflake SCD2 Snap to support Pipeline parameters for Natural key and Cause-historization fields. | |
Nov 23, 2021 | 427patches12999 | Enhanced the Reltio Generic Execute with a new field Empty Responses as Separate Documents that enables the Snap to write empty responses as separate documents. | |
Nov 23, 2021 | 427patches12999 | Revised the field labels in Redshift Cross-Account IAM Role Account from:
|
Platform
New Feature
Custom Org Branding: SnapLogic administrators can now customize the appearance of their SnapLogic platform orgs. The new Configure for Org tab in Manager enables you to brand your SnapLogic platform, including adding a custom logo and background color for each org.
After you upload the desired logo and set the background color for each Org, SnapLogic displays your Org branding logo and background color changes:
Manager Toolbar: Added the Flows tab in the Manager Toolbar. The tab contains the Flows or Pipelines that you create in the Flows application and save them in the shared project space. To know more about Flows, see Introducing SnapLogic Flows.
Enhancements
Snaplex Configuration: Increased the maximum memory allocation for the heap size auto setting.
For RAM up to 4GB, 75% is allocated to JVM.
For RAM over 4GB and up to 8GB, 80% is allocated to JVM.
For RAM over 8GB and up to 32GB, 85% is allocated to JVM.
For RAM over 32GB, 90% is allocated to JVM.
This change will take effect after the Monitor process restarts. To get the updated behavior on Linux Groundplex instances, restart the JCC using jcc.sh restart
. On Windows, if running as a service, update the service using jcc.bat update_service
, otherwise run jcc.bat restart
. On Cloudplex instances, the change takes effect automatically.
Snaplex Logging: Increased the maximum log file size from 50 megabytes to 300 megabytes to accommodate larger log files.
Task Details Page: The Unsecure URL field is renamed to Unsecured URLs and the Local Override/Override Secure URL field is renamed to Snaplex URL.
Pipeline Execution Data: Increased the child Pipeline execution retention limit from last 100 to last 500 on the Dashboard to assist with monitoring and troubleshooting.
Known Issues
Generic JDBC account validations fail if the account is not used in a global shared folder.
The Help icon that leads to the SnapLogic documentation site is disabled by default for unsaved Snaps. Save the Snap to enable the Help icon.
Behavior Change
Cloud Pattern Catalog: Enhanced the Pipeline pattern search to find patterns by endpoints, technology, and use cases. It also supports both full and partial keyword search. In the search results, hovering over any pattern now displays its description.
Executions created after this release will be available in the SnapLogic Dashboard for 45 days. For Ultra tasks, the executions will be retained for 15 days after completion. Complete execution data will still be available in the Runtime Archive in SnapLogic Manager.
Metadata Snaps now validate Pipeline fields against the schema. If a SnapLogic Create Snap or a SnapLogic Update Snap would modify an existing Pipeline schema, an error is thrown to avoid affecting downstream processes that depend on the same Pipeline schema.
API Management
New Feature
API Specifications Editor: Added the capability to edit API specifications from the publish dialog window.
Introducing the API Subscription Manager
Added a new page, Subscription Manager, which allows Org administrators to manage user subscription applications and user sign-up requests. This page has the following tabs:
User Access: Approve new user sign-up requests.
Applications: Approve applications and grant or revoke access to subscriptions.
Behavior Change
API Management Generic OAuth2 Policy: Added a new required field OAuth State: Time-To-Live in Seconds that determines the number of seconds for which the OAuth state is active.
Developer Portal Enhancements
Added featured APIs to the Developer Portal. An API can be marked “featured” to promote the API in the developer portal. An applications page in the portal lets users create or manage applications and API subscriptions.
Developer Portal Login: Added user account creation, login, and profile management for the Developer Portal. Users get access to the Developer Portal after the Org Administrator approves their account.
Notifications: Users will now receive email notifications upon account approval, password reset, and subscription updates.
Known Issue
User accounts created in the Developer Portal do not show up in the User Access tab of the Subscription Manager. To avoid this issue, follow the steps to Create a New Portal User.
Snaps
New Snaps
Amazon SNS: Amazon Simple Notification Service (Amazon SNS) is a fully managed service that provides message delivery from publishers to subscribed endpoints. This Snap Pack enables you to:
Create, manage and subscribe to topics.
Manage, confirm, and delete subscriptions of users.
Publish messages.
Amazon SNS Snap Pack has the following Snaps:
Create Topic: Creates an Amazon SNS topic - a logical access point that acts as a communication channel. A topic lets you group multiple endpoints.
Manage Topic: Updates and remove attributes and tags in a topic.
Subscribe Topic: Subscribes an endpoint to a topic. The endpoint begins to receive messages published to the associated topic.
Delete Topic: Deletes an Amazon SNS topic with the help of the SNS topic Amazon Resource Names (ARN).
Confirm Subscription: Verifies if the subscription of an endpoint is confirmed.
Delete Subscription: Deletes a subscription. You can delete only the confirmed subscriptions.
Publish Message: Publishes messages to a topic. When a message is published, Amazon SNS attempts to deliver the message to the subscribed endpoints.
Set Subscription Attribute: Allows a subscription owner to set an attribute of the subscription to a new value.
Snap Enhancements
Active Directory: Enhanced the Active Directory Search Snap with a new field set, Returning Attributes, to define attributes that you want the Snap to return in the search results.
Enhanced the File Operation Snap with a new field Azcopy absolute path that enables the Snap to make a single API call to move an entire directory. This field supports directory and wildcard characters for Azure blob storage for all three operations: Copy, Move and Rename.
Enhanced the Azure Storage Account with a new option called SAS URI in the Auth Type dropdown. This enables you to select either Access Key or the SAS URI while configuring the account settings.
Box: Added Connection timeout and Read timeout fields in Box accounts that enable the Box Snaps to read and write data within the specified connection and read timeouts.
Flow:
Enhanced the Gate Snap with the Output Document Content field to indicate the count of input documents and the specific (first or last) document to include in the output.
Enhanced the Filter Snap with the Null-safe access checkbox. Select this checkbox to consider the missing data or empty value as null when accessing the source path.
Google BigQuery: Enhanced the Google BigQuery Bulk Load (Cloud Storage) Snap with the following batching and retry properties to process input records:
Batching: Processes the input records in batches.
Batch Size: The number of records batched per request.
Batch Timeout (milliseconds): Time in milliseconds to elapse following which the batch, if not empty, will be processed even though it might be lesser than the given batch size.
Hadoop: Enhanced the Parquet Writer and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration.
ML Data Preparation: Enhanced the Type Converter Snap with the Fail safe upon execution checkbox. Select this checkbox to enable the Snap to ignore invalid data types and convert data with valid data types.
Oracle: Enhanced the Oracle Bulk Load Snap with Additional SQL Loader Parameters field set to define additional SQL Loader parameters if required.
OpenAPI and REST: Enhanced OpenAPI OAuth2 and REST OAuth2 accounts with a new option password for Grant Type to obtain an access token through user credentials. On selecting password, Snap populates the Username and Password fields to enter the credentials.
RabbitMQ: Enhanced the Rabbit MQ Account with a new checkbox, Cluster Support, which enables multiple cluster node support for RabbitMQ Consumer and RabbitMQ Producer Snaps. You can select this checkbox to enable the high availability of clusters by adding multiple hosts (Hostname and Port) for each cluster node.
Added Salesforce Mutual Authentication Account to Salesforce Snap Pack that provides an additional layer of security while authentication. Both the client and server authenticate themselves using Client Certificate Authentication. This account is not supported for Salesforce Subscriber and Salesforce Publisher Snaps.
Extended the support of Salesforce OAuth2 Account to all the Salesforce Snaps in addition to the Salesforce Bulk Snaps.
Revised the name of Salesforce Attachment Download to Salesforce Download Snap. Enhanced this Snap with a new field File Type to support download of attachments from Salesforce by passing Content Data ID in addition to the Attachment ID.
Updated the default Salesforce API Service Version to 52.0 for the following Snaps.
Enhanced the following Snaps to support Pipeline parameters only for expression fields, while the input data parameters are not supported for expression fields.
Snowflake: Added the following truncate options to Snowflake - Bulk Load Snap to truncate existing data before performing data load to more efficiently transfer data where possible. .
Truncate data: Truncates existing data before performing data load.
Truncate columns: Truncates column values that are larger than the maximum column length in the table.
Added dynamic memory management to the Group By Fields Snap, which can activate with the following settings:
Memory Sensitivity: The Snap's response to the memory changes.
Min Part Size: The minimum part size that you want the Snap to split larger groups into multiple parts.
Enhanced the JSON Formatter, JSON Generator, and JSON Parser Snaps with the Support Type Extensions checkbox. On selecting this checkbox, the Snap supports SnapLogic-specific syntax for representing non-standard data types such as byte arrays and date objects in JSON.
Added the following checkboxes in the Excel Multi Sheet Formatter Snap:
Convert numeric string to number: Sets the behavior of the Snap when a value in the input document is a numeric string. This checkbox supports the conversion of numeric values to string values.
Translate date and time types: Sets the behavior of the Snap when a value in the input document is a date and datetime type. This checkbox supports translating date and time types to Excel date numeric type with date format style.
Workday: Enhanced the Workday Read Snap with the new field Pool Size that controls the maximum number of threads in the pool. This field is available only when you select the Multi-threaded checkbox.
Enhanced the following SQL-based accounts to skip batch execution, when the Batch size value is one. When the Batch size is greater than one, the batch is executed.
Enhanced the following Database Execute Snaps to invoke stored procedures.
Renamed Snap
Salesforce: Renamed the Salesforce Attachment Download Snap to Salesforce Download Snap. This does not affect your subscription to this Snap Pack.
Known Issues
File Operation Snap: Azure storage SDK does not support the file system for
https
. When you do not provide Azcopy path and the Source/Target URL ishttps
, the Snap displays an error. You must provide the SAS URI (besides the container URL of the Azure blob) under the Advanced properties field-set for successful authentication.File Writer Snap does not fail and turns green after execution despite providing an expired SAS URI. Select Validate after write to fail the Snap in case of invalid credentials.
Salesforce Subscriber Snap might duplicate error documents for a single input when the Snap configuration is invalid. For example, invalid Snap settings (Replay ID or Events) or a non-existent event.
Salesforce Mutual Authentication account fails on validation when you create the account for the first time and save it. As a workaround, click Apply and then click Validate for successful validation of the account.
The Help icon that leads to the SnapLogic documentation site is disabled by default for unsaved Snaps. Save the Snap to enable the Help icon.
Behavior Change
Database Execute Snaps
In 4.26, when the stored procedures were called using the Database Execute Snaps, the queries were treated as write queries instead of read queries. So the output displayed
message and status keys
after executing the stored procedure.
In 4.27, all the Database Execute Snaps run stored procedures as read queries. The output now displaysmessage
key, andOUT params of the procedure (if any)
. Thestatus key
is not displayed.If the stored procedure has no OUT parameters then only the message key is displayed with value
success
.
Note: If you have any existing Pipelines that are mapped withstatus key
or previous description, then those Pipelines will fail. Hence, you might need to revisit your Pipeline design.
Generic JDBC Database Account, MySQL Database Account, and Snowflake S3 Database Account
From 4.27 release, the validation message after executing SQL query is different when Batch size is 1.
Till 4.26 release, for invalid test data, the database Snaps displayed the error message as follows:
{error=Batch operation failed, reason=<Reason for Failure>}
For example,{error=Batch operation failed, reason=Data truncation: Data too long for column 'id' at row 1, error code: 1406, SQL state: 22001}
In 4.27 release, when the Batch size is 1, SQL statement is executed as individual statement and the error message displayed is as follows:
{error=SQL operation failed, reason=<SQL Statement and Reason>}]
For example,{error=SQL operation failed, reason=SQL [INSERT into enron.book (id, book) VALUES (?,?)]; Data truncation: Data too long for column 'id' at row 1}]
Prior 4.26 Snap Dot Releases that are Merged into 4.27 GA
Data Automation
SnapLogic's data automation solution speeds up the identification and integration of new data sources, and the migration of data from legacy systems. The solution can automatically detect duplicate, erroneous, or missing data, and identify structures and formats that do not match the data model. Data automation can accelerate the loading and transformation of your data into the data warehouse, speeding up the data-to-decisions process.
New Features in ELT Snap Pack
Starting with the 4.27 release, you can use the Snaps in the ELT Snap Pack to perform ELT operations on Google BigQuery. This is in addition to the existing list of supported target databases—Snowflake, Redshift, Azure Synapse, and Databricks Lakehouse Platform.
ELT Snap Enhancements
Enhanced the ELT Aggregate Snap to support COUNT_IF aggregate function for Redshift and Azure Synapse target databases.
Eliminating duplicates with COUNT_IF aggregate function
Note: Selecting the Eliminate Duplicates checkbox while using COUNT_IF aggregate function does not eliminate duplicate records in case of Snowflake and BigQuery databases, as there is no native support for this feature. However, for Redshift, Azure Synapse, and Databricks Lakehouse Platform, the duplicates are eliminated from the list of records when you select this checkbox for COUNT_IF function.Enhanced the ELT Database Account connecting to a Databricks Lakehouse Platform (DLP) to support two new options - Optimize Write and Auto Compact for creating/replacing a table using any of the ELT Insert Select, ELT Merge Into, ELT Load, and ELT SCD2 Snaps.
Updated the Expressions and Functions Supported for ELT in the Snap and Account configuration sections. This list is common to all target CDWs supported. You can use these expressions to define your Snap or Account settings with the Expression symbol = enabled, where available.
Enhanced the ELT Load Snap to ensure that the Snap uses the default S3 Folder name specified in the Snap's account to accurately resolve the defined File Name Pattern.
Enhanced the ELT Select and ELT Execute Snaps to allow SQL comments inside the SQL Query Editor and SQL Statement Editor fields, respectively.
Enhanced the ELT Transform Snap to display the exact data types of fields listed in the Input Schema and Target Schema in case of Azure Synapse.
Enhanced the ELT Merge Into Snap to support MERGE INTO ALL option and automatic source table aliasing.
Enhanced the ELT Load Snap by adding the Source File to Target Table Columns Map field set to enable mapping of columns between the source file and the target table. In case of Databricks Lakehouse Platform the Snap is enhanced to support delimiters other than the comma in the source CSV files.
Enhanced the ELT Join Snap to support Left Anti and Left Semi join types in BigQuery, though it does not natively support these join types.
Enhanced the underlying load mechanism for ELT SCD2 Snap from Insert-and-Update mode to Merge-into mode to substantially improve the Snap's performance while working with large and very large volumes of data (upwards of 500M rows or 50GB size).
Known Issues
When your Databricks Lakehouse Platform instance uses Databricks Runtime version 8.4 or lower, ELT operations involving large amounts of data might fail due to the smaller memory capacity of 536870912 bytes (512MB), which is allocated by default. This issue does not occur if you are using Databricks Runtime version 9.0.
[Fixed in 427patches13923 on Jan 07, 2022] In case of Databricks Lakehouse Platform, when you specify a CSV file to load data from and select the Load Action as Alter Table in the ELT Load Snap, the Snap fails with the error
Database encountered an error during Bulk Load process
.When you configure an ELT Merge Into Snap to perform an Update or Delete operation or an ELT Execute Snap with a MERGE INTO statement that performs Update or Delete operation on a Databricks Lakehouse Platform cluster, it may return an error if multiple source rows attempt to update or delete the same target row. To prevent such errors, you need to preprocess the source table to have only unique rows.
While using the SQL Expressions and Functions Supported for ELT to define the field values in ELT Snaps and accounts:
DO NOT use regular JavaScript expressions (supported in the Standard-mode Snaps and not in the above list).
DO NOT prefix variable names with $ symbol.
The Help icon that leads to the SnapLogic documentation site is disabled by default for unsaved Snaps. Save the Snap to enable the Help icon.
Prior 4.26 ELT Dot Releases that are Merged into 4.27 GA
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.