Versions Compared

Old Version 241

changes.mady.by.user Sandhya Sriram

Saved on

compared with

New Version Current

changes.mady.by.user Ankur Parekh

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

...

Expand

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT 1: Friday,  – Snaplex version: 4.27_rc-10694 - 4.27

  • UAT 2: Friday,  – Snaplex version: 4.27_rc-10694 - 4.27

  • GA: Saturday,  – Snaplex version: main-10704 - 4.27 GA

The SnapLogic GA release build on the production server is also deployed on the UAT server.

Info

Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the recommended version (main-10919 4.27 Patch 3) at 9 p.m. PT on Saturday, . Orgs migrated to the 4.27 GA version prior to the auto-upgrade are not impacted. 
    The Snaplex upgrade process also upgrades the Java version to the latest version. After the upgrade, the Java version of your Cloudplex nodes may not be the same as your FeedMaster nodes. However, this does not impact the operations of your SnapLogic instance. We will be fixing this issue in a subsequent release.

  • To use the new SnapLogic features and Snaps in the 4.27 release, ensure that your Snaplex nodes are upgraded to the main-10704 (Recommended) version.

  • This 4.27 release includes the patches deployed to SnapLogic after the 4.26 release, as listed in the 4.26 Dot Releases section. 

4.27 Dot Releases

Platform

Log4j upgrade: You need to restart your Snaplex monitor process using jcc.sh restart to update the monitor process and use the latest Log4j version.

Date of Update

Snaplex Build

Update

main-11121 4.27 Patch 4

Upgraded to Log4j version 2.17.1.

main-10919 4.27 Patch 3 (Recommended)

  • Upgraded to Log4j version 2.17.0.

  • Fixed an issue with Patch 2 that could cause some Pipeline Execute Snap instances to hang.

main-10904 4.27 Patch 2

Fixed a potential vulnerability issue regarding Log4j. Upgrading to Log4j 2.16.0 fixed this issue by removing support for message lookup patterns and disabling JNDI functionality by default.

Note

  • Some Pipelines may hang, where the parent and child Pipelines run on nodes having different Snaplex versions, causing the Snaplex to effect a rolling restart. 

  • We plan to release a patch next week to address the above issue.

main-10894 - 4.27 Patch 1

Fixed an issue with the Log4j 2 (Java logging library) that affected Java applications. For additional details, see our Community blogpost.

...

Snap Pack

Date of Update

Snap Pack Version

Updates

Marketo

427patches13147

Fixed an issue with the Asset Read, Lead Read, and Asset Write Snaps where the original payload was not passed through to the error view.

Marketo

427patches12999

Fixed an issue where the Bulk Lead Extract and Bulk Lead Upsert Snaps failed when the Snaplexes were proxy-enabled.

Salesforce

427patches14079

Fixed an issue with the Salesforce Subscriber Snap where the Snap was unable to re-subscribe to a topic if it did not receive any message for 72 hours — it stopped and displayed an error message. With this fix, the Snap automatically sets the Replay ID value to -2 when:

  • the specified Replay ID is invalid or outside the retention window.

  • the CreatedDate of the most recent event received by Snap is outside the 72-hour retention window.

Hadoop

427patches13769

Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM.

Binary

427patches14283

  • Fixed an issue with the Binary Snaps where the Snaps could not re-establish a lost connection to the SMB mount point.

  • Enhanced the S3 File Writer Snap with the Suggest fully-qualified file names checkbox that allows you to use fully-qualified file names in the File name Suggestions list for gov cloud instances.

  • Fixed an issue with the File Poller Snap to handle poor performing polling operations.

  • Fixed an issue with the ZipFile Write Snap where the ZIP extension files were downloaded with an incorrect file extension.

  • Fixed an issue with the Zip File Read Snap where a zip file from SMB failed to uncompress due to the limited compatibility of Java unzip utility, which is nowreplaced with Apache unzip utility.

  • Fixed an issue with the File Writer Snap when a file greater than 10MB failed to write when uploading to Google Storage.

  • Upgraded the Google Cloud Storage Java Library from version 1.105.1 to version 2.2.2.

Amazon SQS

427patches14012

Fixed an issue in the SQS Consumer Snap where the Snap did not read message attributes when the Client-side buffering checkbox is selected.

Azure Service Bus

4.27patch14048

Introduced the Azure Service Bus Snap Pack— a cloud messaging service that connects applications, devices, and services running in the cloud to any other applications or services. This Snap Pack offers the following Snaps:

Transform

427patches12966

Enhanced the Avro Formatter Snap to display meaningful error message while processing invalid and null values from the input.

Google BigQuery

427patches13752

Upgraded Google BigQuery driver to 1.119.0 version to support time partition intervals by MONTH and YEAR.

SAP

427patches13992

  • Upgraded the Java Native Library used by the SAP Java Connector (JCo) to version 3.1.4.

Note

Breaking change for Pipelines containing Snaps from the SAP Snap Pack with Java Native Library

This upgrade may cause your existing Pipelines to break if they contain Snaps from the SAP Snap Pack. You must manually upgrade your SAP Java Connector (JCo) Native Library to version 3.1.4 for these Pipelines to run successfully. See SAP Configurations for more information.

  • Enhanced the SAP Execute Snap to support nested tables or structured data in the BAPI function module's import parameters.

ELT

427patches13923

  • Fixed an issue with the ELT SCD2 Snap where the Snap did not equate null values in the corresponding cause-historization rows of both the source and target tables (with no changes to data in the remaining fields) as the same and produced duplicate rows in the target table, as a result. After this fix, the Snap does not cause any new duplicate rows in the target table.

  • Fixed an issue with the ELT Load Snap where the Snap fails with the error Database encountered an error during Bulk Load process when you specify a CSV file to load data from, with the Load Action as Alter Table. The Snap now performs the specified ALTER TABLE actions—ADD/DROP columns—and loads the data into the target table accordingly (without the need to manually modify the source or target tables beforehand).

ML Core
ML Data Preparation

427patches13948

Fixed an issue with the following Snaps, where a deadlock occurred when data is loaded from both the input views.

ML Analytics

427patches13292

Fixed an issue with the Profile Snap, where the Snap failed to create a temporary directory as the temp file path exceeded 255 characters.

PostgreSQL

427patches13149

Fixed an issue with the PostgreSQL - Execute Snap, where the Snap failed when using the Delete query with the RETURNING function.

Salesforce

427patches13944

Fixed an issue with Salesforce accounts, where an account password containing special characters and an empty security token was improperly encoded, due to which the account was not validated.

NetSuite

427patches13080

Fixed an XML parsing issue with empty elements for the NetSuite Snaps.

Salesforce

427patches13789

Fixed the following issues with the Salesforce Snaps:

  • High memory use due to leaked resources.

  • Hung Pipelines due to blocked threads.

Redshift

427patches12999

Fixed an issue with the Redshift Bulk Load Snap, where the temporary files in S3 were not deleted for aborted or interrupted Pipelines.

Exact Online

427patches (main96)

Introduced the Exact Online Snap Pack that enables businesses to automate operations in a single flexible subscription, improve business results, and maximize data security. This Snap Pack offers the following Snaps:

SAP S/4 HANA

427patches (main95)

  • Enhanced the SAP S/4 HANA Snaps to support sub-entities and pagination support.

  • Removed unnecessary datatype conversions.

  • Reduced metadata request volume.

  • Fixed issues in concurrency exception (exceptions that arise when two users attempt to change the same data in a database simultaneously) and potential Snap crashes.

Google BigQuery

427patches13615

Fixed the table truncate 404 error in the Google BigQuery Load (Streaming) Snap by supporting the retry functionality. The Snap now waits in case of an error and retries before loading the data.

ELT

427patches13539

  • Fixed an issue with ELT SCD2 Snap where the Snap fails when you define more than one TargetTable Natural Key in the Snap configuration to load SCD2 data into the target Cloud Data Warehouse instance.

  • Fixed an issue with the ELT SCD2 Snap where the Snap failed to update the previous current rows in the target SCD2 table to historical rows when you define an End Date of Historical Row in Target Table Temporal Field(s).

  • Fixed an issue with the ELT SCD2 Snap where the Snap failed to insert new rows when you define the values that exist in the most recent historical rows of the target SCD2 table as the cause-historization values.

  • Fixed an issue with ELT Transform Snap where the Snap does not omit the source columns marked for removal from the output view—using an empty Target Path for one or more columns selected in the Expression field of the Snap’s Mapping Table. See Using Empty Target Paths to Omit Rows from the Snap Output to understand how to perform this operation.

Splunk

427patches13429

Updated the Splunk Basic Auth account with the following:

  • Fixed the connection failure issue when connecting to a cloud-based Splunk instance by not adding the prefix 'input-'to the hostname.

  • Removed the On-premises checkbox from the Snap Account settings.

Binary

427patches13004

Fixed an issue with the SMB Account in Binary Snaps where the Snaps could not re-establish a lost connection to the SMB mount point.

NetSuite

427patches13072

Fixed an issue with the NetSuite Search Snap where the Snap failed when the body is not present in the SOAP request.

REST

427patches12750

Enhanced the REST Put Snap with the Upload Transfer Request Type field to support encoding of chunked data transfer and calculating content length.

Anaplan

427patches13160

Fixed an issue with the Anaplan Snaps, where the log messages were stored in jccoutput.log file instead of jcc.json file, and updated the dependent libraries.

Transform

main12989

Fixed an issue with the Group by Fields Snap that caused the Snap to abort with an error when executed with zero input documents. 

REST

OpenAPI

main12989

Fixed an issue with the REST OAuth2 and OpenAPI OAuth2 accounts that caused a new feature, OAuth Password Grant Type, to be excluded from the 4.27 GA version.

ELT

427patches13030

  • Fixed the following issues with the ELT SCD2 Snap:

    • Where the Snap failed to get the right data type due to column name case mismatches between what is used in the Snap and what is actually used in the Azure Synapse tables (returned by the JDBC driver). You no longer need to type the column names in the exact case that Azure Synapse expects.

    • The Snap failed with the error start_date does not exist while writing SCD2 data to a Redshift table column start_date that is specified as the Start Date of Current Row in the Target Table Temporal Field(s) field set.

    • The Snap failed with the error Reference 'END_DATE' is ambiguous while merging SCD2 updates into DLP tables.

    • Where the Snap failed due to lack of required access privileges on the target database (for example, create table rights to create temporary tables as needed). The Snap now runs the input SQL statement and the elaborate sub-queries instead of attempting to create a temporary table in such scenarios.

  • Fixed the issue with the ELT SCD2 and ELT Load Snaps that fail to perform an add/drop operation involving multiple columns on a Redshift target table.

SAP

427patches12999

Enhanced SAP RFC Listener, SAP IDoc Listener and SAP IDoc Document Listener Snaps with a new field set Advanced JCo Server Properties that allows you to configure Java Connector (JCo) server-related attributes.

Google BigQuery

427patches12691

Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap, where the Snap failed with an exception for big query tables. The CreateDisposition is now set conditionally on the basis of the setting in Create table if not present checkbox.

Hadoop

427patches12999

Enhanced the Parquet Reader Snap with int96 As Timestamp checkbox, which when selected enables the Date Time Format field. You can use this field to specify a date-time format of your choice for int96 data-type fields. The int96 As Timestamp checkbox is available only when you deselect Use old data format checkbox.

Snowflake

427patches12999

Enhanced the Snowflake SCD2 Snap to support Pipeline parameters for Natural key and Cause-historization fields.

Reltio

427patches12999

Enhanced the Reltio Generic Execute with a new field Empty Responses as Separate Documents that enables the Snap to write empty responses as separate documents.

Redshift

427patches12999

Revised the field labels in Redshift Cross-Account IAM Role Account from:

  • Cross-Account IAM Role ARN (Writing to Bucket) to S3 Bucket Write IAM Role ARN

  • Cross-Account IAM properties (S3 Bucket) to S3 Bucket Read IAM Role.

  • Removed Key/Trust Store Password and Key Alias fields in Redshift SSL and Redshift Cross-Account IAM Role SSL accounts.

...

Tip

Snaplex-based Scheduler is now enabled by default

During the 4.26 release cycle's auto-upgrade date of September 18, the Snaplex-based Scheduler /wiki/spaces/AP/pages/1754562858 was enabled by default through the Control Plane. The new scheduling mechanism improves the timeliness and reliability of Scheduled Task executions. You do not need to update your Snaplex instances to enable the new scheduler, unless you have set the feature flag to false for your Org, in which case you will have to switch it to true in order for the Snaplex-based Scheduler to be enabled for your Org. Contact support@snaplogic.com if you are experiencing any issues with your Scheduled Tasks. 

...

  • Custom Org Branding: SnapLogic administrators can now customize the appearance of their SnapLogic platform orgs. The new Configure for Org tab in Manager enables you to brand your SnapLogic platform, including adding a custom logo and background color for each org.
    After you upload the desired logo and set the background color for each Org, SnapLogic displays your Org branding logo and background color changes:

...

  • Manager Toolbar: Added the Flows tab in the Manager Toolbar. The tab contains the Flows or Pipelines that you create in the Flows application and save them in the shared project space. To know more about Flows, see Introducing SnapLogic Flows.

...

Enhancements

Snaplex Configuration: Increased the maximum memory allocation for the heap size auto setting.

  • For RAM up to 4GB, 75% is allocated to JVM.

  • For RAM over 4GB and up to 8GB, 80% is allocated to JVM.

  • For RAM over 8GB and up to 32GB, 85% is allocated to JVM.

  • For RAM over 32GB, 90% is allocated to JVM.

...

Box: Added Connection timeout and Read timeout fields in Box accounts that enable the Box Snaps to read and write data within the specified connection and read timeouts.

Flow:

  • Enhanced the Gate Snap with the Output Document Content field to indicate the count of input documents and the specific (first or last) document to include in the output.

  • Enhanced the Filter Snap with the Null-safe access checkbox. Select this checkbox toconsider the missing data or empty value as null when accessing the source path.

Google BigQuery: Enhanced the Google BigQuery Bulk Load (Cloud Storage) Snap with the following batching and retry properties to process input records:

  • Batching: Processes the input records in batches.

  • Batch Size: The number of records batched per request.

  • Batch Timeout (milliseconds): Time in milliseconds to elapse following which the batch, if not empty, will be processed even though it might be lesser than the given batch size.

Hadoop: Enhanced the Parquet Writer/wiki/spaces/SD/pages/1438227 and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration. 

...

...

  • Added dynamic memory management to the Group By Fields Snap, which can activate with the following settings: 

    • Memory Sensitivity: The Snap's response to the memory changes. 

    • Min Part Size: The minimum part size that you want the Snap to split larger groups into multiple parts. 

  • Enhanced the JSON Formatter, JSON Generator, and JSON Parser Snaps with the Support Type Extensions checkbox. On selecting this checkbox, the Snap supports SnapLogic-specific syntax for representing non-standard data types such as byte arrays and date objects in JSON.

  • Added the following checkboxes in the Excel Multi Sheet Formatter Snap:

    • Convert numeric string to number: Sets the behavior of the Snap when a value in the input document is a numeric string. This checkbox supports the conversion of numeric values to string values.

    • Translate date and time types: Sets the behavior of the Snap when a value in the input document is a date and datetime type. This checkbox supports translating date and time types to Excel date numeric type with date format style.

Workday: Enhanced the Workday Read Snap with the new field Pool Size that controls the maximum number of threads in the pool. This field is available only when you select the Multi-threaded checkbox.

Enhanced the following SQL-based accounts to skip batch execution, when the Batch size value is one. When the Batch size is greater than one, the batch is executed.

...

Renamed Snap

Salesforce: Renamed the Salesforce Attachment Download Snap to Salesforce Download Snap. This does not affect your subscription to this Snap Pack.

Known Issues

  • File Operation Snap: Azure storage SDK does not support the file system for https. When you do not provide Azcopy path and the Source/Target URL is https, the Snap displays an error. You must provide the SAS URI (besides the container URL of the Azure blob)under the Advanced properties field-set for successful authentication.

  • File Writer Snap does not fail and turns green after execution despite providing an expired SAS URI. Select Validate after write to fail the Snap in case of invalid credentials.

  • Salesforce Subscriber Snap might duplicate error documents for a single input when the Snap configuration is invalid. For example, invalid Snap settings (Replay ID or Events) or a non-existent event.

  • Salesforce Mutual Authentication account fails on validation when you create the account for the first time and save it. As a workaround, click Apply and then click Validate for successful validation of the account.

  • The Help icon (blue star) that leads to the SnapLogic documentation site is disabled by default for unsaved Snaps. Save the Snap to enable the Help icon.

...

New Features in ELT Snap Pack

Starting with the 4.27 release, you can use the Snaps in the ELT Snap Pack to perform ELT operations on Google BigQuery. This is in addition to the existing list of supported target databases—Snowflake, Redshift, Azure Synapse, and Databricks Lakehouse Platform.

Note

Support for ELT operations in Google BigQuery

ELT Snap Pack does not support the Legacy SQL dialect of Google BigQuery. So we recommend that you use the BigQuery Standard SQL dialect across ELT Snaps.

...

  • Enhanced the ELT Aggregate Snap to support COUNT_IF aggregate function for Redshift and Azure Synapse target databases.
    Eliminating duplicates with COUNT_IF aggregate function
    Note: Selecting the Eliminate Duplicates checkbox while using COUNT_IF aggregate function does not eliminate duplicate records in case of Snowflake and BigQuery databases, as there is no native support for this feature. However, for Redshift, Azure Synapse, and Databricks Lakehouse Platform, the duplicates are eliminated from the list of records when you select this checkbox for COUNT_IF function.

  • Enhanced the ELT Database Account connecting to a Databricks Lakehouse Platform (DLP) to support two new options - Optimize Write and Auto Compact for creating/replacing a table using any of the ELT Insert Select, ELT Merge Into, ELT Load, and ELT SCD2 Snaps.

  • Updated the Expressions and Functions Supported for ELT in the Snap and Account configuration sections. This list is common to all target CDWs supported. You can use these expressions to define your Snap or Account settings with the Expression symbol = enabled, where available.

  • Enhanced the ELT Load Snap to ensure that the Snap uses the default S3 Folder name specified in the Snap's account to accurately resolve the defined File Name Pattern.

  • Enhanced the ELT Select and ELT Execute Snaps to allow SQL comments inside the SQL Query Editor and SQL Statement Editor fields, respectively.

  • Enhanced the ELT Transform Snap to display the exact data types of fields listed in the Input Schema and Target Schema in case of Azure Synapse.

  • Enhanced the ELT Merge Into Snap to support MERGE INTO ALL option and automatic source table aliasing.

  • Enhanced the ELT Load Snap by adding the Source File to Target Table Columns Map field set to enable mapping of columns between the source file and the target table. In case of Databricks Lakehouse Platform the Snap is enhanced to support delimiters other than the comma in the source CSV files.

  • Enhanced the ELT Join Snap to support Left Anti and Left Semi join types in BigQuery, though it does not natively support these join types.

  • Enhanced the underlying load mechanism for ELT SCD2 Snap from Insert-and-Update mode to Merge-into mode to substantially improve the Snap's performance while working with large and very large volumes of data (upwards of 500M rows or 50GB size).

...