Per the SnapLogic Release Process, all remaining Snaplex instances across customer Orgs are upgraded to the recommended 4.21 version (master-7451 - 4.21 Patch 1)at 9 p.m. PT on June 13, 2020. If you upgraded your Snaplex to a 4.21 version prior to the auto-upgrade event, then your Snaplex version does not change.
To use the new features and Snaps in the 4.21 release, ensure that your Snaplex nodes are upgraded to the recommended 4.21 patch version (master-7451 - 4.21 Patch 1).
This 4.21 GA release includes the patches deployed to SnapLogic after the 4.20 GA release, as listed in the 4.20 Dot Releases page.
We recommend that you upgrade to the latest Chrome or Firefox browser version and clear your browser cache before logging into the SnapLogic Intelligent Integration Platform (IIP).
Java 11 Support
Starting from the 4.18 August release, SnapLogic supports Java 11 for Groundplex instances. We recommend all customers to migrate to Java 11, since Oracle no longer releases security updates for Java 8. For details, see Configuring the Java Version on Your Groundplex.
Groundplex:
The 4.21 Snaplex installer uses Java 11 by default. Java 8 is also supported as a rollback option, if required.
Current Groundplex installations will continue to work with Java 8 until the Snaplex is reinstalled using the new installer.
You are required to upgrade their Groundplex instances to Java 11 using the new installer. Support for Java 8 will cease from the August 2020 release.
Cloudplex:
During the current 4.21 release, all your existing Cloudplex instances hosted in the United States will be migrated to Java 11. Currently, this is being planned for Friday, June 26, 2020. For more details, see Migrating Your Cloudplex Nodes to Java 11.
New IP Whitelisting Requirements Reminder
We remind all customers of the new IP Whitelisting requirements that have been in effect since the August 2019, 4.18 release. Ensure that your network administrators review the following:
Insights dashboard: Revamps the Insights dashboard to provide data visualization of Org-level Pipeline-execution statistics for a specific date range.
Task monitoring: Adds a new tab for Tasks under Dashboard that does data visualization of triggered and scheduled tasks for a selected time range. The new tab displays the number of failed and completed executions for each task type along with their details.
Snap auto-configuration: Enhances the suggestion capabilities of Iris to include Snap auto-configuration in Snaps with suggestible properties. Iris Expression recommends the configuration of these properties based on similar Snap settings from Pipelines that you can access.
Sticky Notes in SnapLogic Designer: Introduces the ability to add notes to a Pipeline in SnapLogic Designer. This enables you to collaborate with multiple Pipeline developers or add important notes for complex Pipelines. The notes are included in the Pipeline metadata and are immediately visible to other users accessing the Pipeline.
When you copy a Pipeline with Sticky Notes, the notes are also copied with the Pipeline. Additionally, when you modify or add Sticky Notes in the copied Pipeline, the notes in the original Pipeline are also updated.
Target recommendation: Enhances the Iris Expression recommendation capabilities to include Target path suggestions in Mapper Snap. Iris recommends Target paths based on exact matches between Expression and Target paths from other Pipelines in your Org.
Known Issues
A Triggered Task on a Groundplex with an underlying Pipeline that contains child Pipelines might stall as the Data Validator Snap is executing in the parent Pipeline.
An Ultra Pipeline sometimes processes the same message twice even though the message was processed successfully during the first attempt.
Updates to Expression Behavior
Starting in the 4.21 Release (May 2020), the escaping required in the replacement text of the ReplaceAll function is identical to that of the Replace function. Prior to this release, you had to use additional escaping in the replacement text with ReplaceAll to get the desired output.
Previous Behavior
New Behavior
Example: $msg.replaceAll("-", "\\\\-")
where $msg contains mother-in-law
Result: mother\-in\-law
Example: $msg.replaceAll("-", "\\-")
where $msg contains mother-in-law
Result: mother\-in\-law
Starting in the 4.21 Release (May 2020), the Date.parse() function returns the response text Not a Number when you enter a non-number or infinity. Prior to this release, doing either resulted in an error, even though the formatting of the string was correct.
Previous Behavior
New Behavior
Example: $date.parse(Math.abs(1/0))
where $date is infinity
Result: 1970-01-01T00:00:00.000 UTC
Example: $date.parse(“foo”, “yyyymmdd”)
Result: Error
Example: $date.parse(Math.abs(1/0))
where $date is infinity
Result: Not a number
Example: $date.parse(“foo”, “yyyymmdd”)
Result: Not a number
Standard-mode Snaps
New Snaps
SharePoint Online Snap Pack: This Snap Pack enables organizations to share and manage content and collaborate across an organization using the following SharePoint Online Snaps:
SharePoint Online - Search Items: Searches for the hierarchy of items matching the search query within a folder hierarchy, a whole drive, or files shared with the user.
SharePoint Online - List Read: Reads metadata of a list along with its columns and items from the SharePoint Online site.
SharePoint Online - Upload File: Uploads the contents of a new file or updates the contents of an existing file in a document library.
Adobe Experience Platform Snap Pack: This Snap Pack enables you to integrate with the new Adobe Experience Platform and includes the following Snaps:
Adobe Experience Platform File Generator: Generates an Adobe Enterprise Platform (AEP)-compatible file by restructuring upstream data to match a specific schema stored in AEP.
Adobe Experience Platform S3 Connector:
Enhances theSnap by removing theCreate New Datasetcheckbox. The Snap now automatically creates a new dataset if it does not exist.
Retrieves data from AWS S3 and writes it into AEP.
Cassandra: Extends support to Cassandra version 3.11.5.
Adobe Experience Platform Write: Writes data from upstream Snaps into AEP.
SAP S/4HANA Snap Pack: This Snap Pack supports the SAP S/4 HANA on-premise environment and includes the following Snaps:
ML Data Preparation: Introduces the Mask Snap that enables you to hide sensitive information in your dataset before exporting the dataset for analytics or writing the dataset to a target file.
Updated Snap Packs
Adobe Experience Platform Snap Pack: Enhances theAdobe Experience Platform S3 Connector Snap by removing theCreate New Datasetcheckbox. The Snap now automatically creates a new dataset if it does not exist.
Cassandra: Extends support to Cassandra version 3.11.5.
Confluent Kafka: Enhances the Kafka Consumer Snap to significantly reduce the volume of commit requests for offsets when using the record acknowledgment mode (Wait after each record), and thereby, improve performance and stability.
Enhances the Match Snap to add a new field, Match all, which matches one record from the first input with multiple records in the second input. Also, enhances the Comparatorfield in the Snap by adding one more option, Exact, which identifies and classifies a match as either an exact match or not a match at all.
Enhances the Deduplicate Snap to add a new field, Group ID, which includes the Group ID for each record in the output. Also, enhances the Comparator field in the Snap by adding one more option, Exact, which identifies and classifies a match as either an exact match or not a match at all.
Enhances the Sample Snap by adding a second output view which displays data that is not in the first output. Also, a new algorithm type, Linear Split, which enables you to split the dataset based on the pass-through percentage.
Removes the now redundant Custom WSDL URL Domain field from the NetSuite Token Account to support NetSuite's transition to automatic account-specific domain identification.
SAP: Enhances the SAP Execute Snap to add a new field, Output date timeas Text, which enables you to output date and time as a String data type instead of a DateTime object.
Script Snap Pack: Adds a new Cleanup method in the ScriptHook interface associated with the Script Snap. This method enables the Script Snap to automatically execute a clean-up once the configured script completes executing.
ServiceNow: Adds support for ServiceNow Madrid, New York, and Orlando versions.
Transform: Adds support in the Mapper Snap to display schemas with complex nesting. For example, if Snaps downstream from the Mapper Snap have custom fields that contain multiple keys, then these keys automatically display in the Mapper Snap's Target Schema dropdown list when you toggle-off Expression Builder and select the field in the Mapping Table.
The HTTP header field's behavior in the SOAP Execute Snap has been changed. Specifying key alone is not sufficient. Both key and value must be provided when invoking a task.
Known Issues
In some Snaps, if you leave any expression-enabled mandatory fields empty, a null pointer exception error message might display instead of the usual error message on validation.
The Cassandra Snap Pack does not support the following data types introduced in Apache Cassandra V3.x:
DateRange
Duration
Geo-spatial data types such as Point, Polygon and LineString
Snaps in this Snap Pack display an exception when you query a map column that has a timestamp as the key.
Google Big Query Snap Pack: The Snaps no longer return the affectedRowCount field with a null value for DML (data manipulation language) queries. The said field no longer exists.
This happens due to a known issue in the MySql Database. For more information, see MySQL Bug #65378.
Parquet Writer(Behavior Change): In the 4.20 Snap version (snapsmrc535 and hadoop8776) when a value for a field defined as repeated (for example, devops in the below example) is not present in the input document, the Parquet Writer Snap displays "The required field 'devops' is missing from a document" error. However, in the 4.21 release, the Parquet Writer Snap outputs a null value when there is no input for the repeated fields. If this behavior change impacts your existing Pipelines having the Parquet Writer Snap, then you might want to consider potentially redesigning your impacted Pipelines.
repeated group devops {
required binary lan (UTF8);
optional binary lan2 (UTF8);
}
Custom Snaps
During the May2020, 4.21 release cycle, we did multiple enhancements to the third-party dependencies used by the SnapLogic platform and the Snaps. Hence, we strongly recommend updating and redeploying all your custom Snap Packs to ensure compatibility with this release. For details, seeUpdating Custom Snap Packs in the SnapLogic Developer Documentation.
Prior 4.20 Snap Dot Releases in 4.21 GA
The following 4.20 Snap Dot Releaseupdates are now a part of the current 4.21 GA release.
Fixes an issue in the JMS Consumer Snap due to which the Snap does not release the memory even after Pipeline termination, leading to huge memory spikes.
Fixes the Workday Write Snap that ignores theImport Synchronizedcheck box selection while importing object data. The Snap now:
Waits for the current import request to complete, whether successfully or with error, before initiating the next import request.
Provides accurate real-time status of the request to downstream Snaps.
Existing Pipelines using this Snapmay experience longer execution timesdue to the synchronous behavior. However, you no longer need to use an additional Snap to capture and pass on the request status to downstream Snaps.
Fixes an issue with the Email Sender Snap that caused the Snap to fail while uploading attachments to SMB locations when the attachment password contained a forward slash "/".
Introduces the SAP S/4HANA Snap Pack to support the SAP S/4 HANA Cloud and SAP S/4 HANA On-premise environments. The Snap Pack includes the following Snaps:
Certifies the Snowflake Snap Pack against JDBC Driver version 3.12.3.
With the updated JDBC driver (version 3.12.3), the Snowflake Execute and Multi-Execute Snaps' output displays a Status of "-1" instead of "0" without the Message field upon successfully executing DDL statements. If your Pipelines use these Snaps and downstream Snaps use the Status field's value from these, you must modify the downstream Snaps to proceed on a status value of -1 instead of 0.
This change in the Snap behavior follows from the change introduced in the Snowflake JDBC driver in version 3.8.1: "Statement.getUpdateCount() and PreparedStatement.getUpdateCount() return the number of rows updated by DML statements. For all other types of statements, including queries, they return -1."
Updates the NetSuite Create, Update, Upsert, and Async Upsert Snaps to use the Classificationtype instead of Class, to resolve schema generation issues. While theClasstype remains available in the NetSuite Search and Async Search Snaps for backwards compatibility, use of the Classificationtype is recommended.
Fixes the Active Directory Snaps to successfullyread special charactersin theExisting distinguished name and Distinguished name Snap settings without requiring escape characters.
Adds theUser-defined object metadataandObject tagsproperties in the S3 File Writer Snap and the Get Object Tagsproperty in the S3 File Reader Snap to allow inclusion of user defined metadata and object tags.
Fixes a regression caused by Patch binary8775 wherein the File Reader and S3 File Reader Snaps fail to read S3 files with an account which does not include GetObjectTagging permission.
Fixes the MySQL - Bulk Load Snap to use the UTF-8 character set for writing temporary files in order to successfully handle special characters, irrespective of the value specified in theCharacter set field in the Snap settings.
Updates the Hadoop Snap Pack to use the latest version of org.xerial.snappy:snappy-javafor compression type Snappy, in order to resolve thejava.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I error.
Fixes the Generic JDBC Execute Snap that adds the value of the previous column's field to a NULL field when SELECT queries having duplicate column names are run on the PostgreSQL database.
Fixes the File Writer Snap to allow the deletion of temporary Google Storage files from the/tmpdirectory of the Groundplex if Pipeline execution stops before completion.
Fixes an issue with the CSV Parser Snap wherein the Snap fails when it is configured in the new Form UI if the Contains headers and Validate headers fields are not selected but empty rows exist in the Column names field.
Fixes the JSON Formatter Snap to filter the input data based on the value specified in the Content field in the Snap settings when the JSON lines field is also selected. Previously, the Snap was writing the entire input data to the output file
The JSON Formatter Snap output now includes only those fields from the input file that are specified in the Content field under Settings.
If your Pipelines use the JSON Formatter Snap with the JSON lines field selected, they may fail to execute correctlyif the Content field mentions a specific object or field but the downstream Snap is expecting the entire data. Hence, for backward compatibility, you must review the entries in the Content field based on the desired output, as follows:
Enter $ to include all the fields in the output. OR
Enter or select the specific fields to restrict the output to.
The behavior of the Snap when the JSON lines field is not selected is correct and remains unchanged.
Fixes an issue in the XML Generator Snap due to which the custom XML data in the Edit XML field got ignored when inbound schema and root XML element were also provided, and instead, the output was generated from upstream data. For Pipelines with a standalone XML Generator Snap, a "Premature end of file"error was displayed.
The XML Generator Snap now gives precedence to any custom XML data that is provided over data coming from upstream Snaps,to generate the output.
Existing Pipelines using the XML Generator Snap may fail in the following scenarios. Use the resolution provided to update the Snap settings based on the XML data you want to pass to downstream Snaps.
Breaking Scenario
Resolution
Downstream Snaps expect XML output from upstream Snaps, but custom XML data exists in the XML Generator Snap.
To use custom XML data, ensure that the Inbound schema and XML root element are specified and custom XML data is entered using the Edit XML option.
To use data from upstream Snaps, ensure that theInbound schemaand XML root element fieldsare blank and no custom XML data exists in the XML editor.
In the XML Generator Snap settings, the XML root element and Inbound schema are specified but no custom XML data is provided. This will generate a validation error.
In the XML Generator Snap settings, custom XML data is provided and Validate XML check box is selected, but XML root element and Inbound schema are not specified. This will generate a validation error.
Re-release of fixes from db/snowflake8687 for 4.20: Fixes the Snowflake Bulk Load Snap wherein the Snap fails to load documents containing single quotes when the Load empty strings checkbox is not selected.
Fixes an issue with the Box Snap Pack wherein the OAuth2 token auto-refresh fails for Box accounts. The fix includes a patch released by Box to restore functionality to the "https://www.box.com/api/oauth2/token" URL of the deprecated version 1 of the Box API.
Box recommends useUnlinkrs to start using the endpoint in version 2 of the Box API: "https://api.box.com/oauth2/token". For more information, see the Box blogpost.
Fixes the error that occurs while transforming an XML document into string by retaining only one version of Saxon library – net.sf.saxon:Saxon-HE:9.6.0-10 across the Transform Snap Pack.
Fixes null handling in the Aggregate Snap whereby changing the order of entries in theAggregate fields field set results in different outputs when the input documents contains one or more null values.
Fixes the issue with the Excel Parser Snap whereby the Snap does not parse the date and time correctly for custom date formats.
Fixes Azure Active Directory Snaps where the Snaps fail after running approximately an hour when the access token times out, and the Snap fails to detect the updated token.
Fixes the ZipFile Write Snap where the Snap generates files with incorrect names when the Use input view label checkbox is selected.
Fixes the File Writer Snap where the Snap fails intermittently when there is a network disruption.
Fixes the File Reader Snap where the Snap generates an output view, along with the error view, if there is an error when using the Google Service account.
Fixes the DynamoDB Scan Snap and the DynamoDB Update Snap where:
The DynamoDB Scan Snap is unable to produce a preview output on validation. (Instead, the Snap passes the output to the downstream Snap.)
The DynamoDB Update Snap produces two error documents for the same error during preview output.
eXtreme(Big Data)
New Features and Enhancements
Cross Account IAM Role Support: Users can now specify cross-account IAM role for an AWS account to spin up an eXtremeplex on AWS EMR without needing to share secret and access keys of their AWS accounts.
Enhanced Account Encryption: Introduces Enhanced Account Encryption (EAE) support for AWS accounts. This enables you to encrypt and secure account credentials that access endpoints from SnapLogic using a private key/public key model. The data is encrypted with a public key before it leaves the browser, then decrypted with a private key on the eXtremeplex.
Publicly Available Datasets: Provides free limited datasets in our public AWS S3 bucket and Azure Databricks clusters. You can use these datasets to try out theeXtreme use case Pipelines. Alternatively, you can also generate your own TPC-DS benchmark datasets to use with these Pipelines.
New Snaps
Spark SQL 2.X Snap Pack: Enhances the Snap Pack to support Java Database Connectivity (JDBC). This enhancement adds the following Snaps and account type:
JDBC Insert: Inserts data into a target table through a JDBC connection.
JDBC Select: Fetches data from a target table through a JDBC connection.
eXtreme Execute Snap Pack: The Spark job continues being in the running state even after the SnapLogic Designer/Dashboard displays the Pipeline status as failed on AWS EMR.
Prior 4.20 eXtreme Dot Releases in 4.21 GA
The following 4.20 eXtreme (Big Data) Dot Release update is now a part of the 4.21 GA release.
4.21 UAT Delta
This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The schedule is:
UAT #1: Friday, April 24, 2020 (Release updates are published above)
UAT #2: Friday, May 01, 2020
GA: Saturday, May 09, 2020 (9 p.m. PT)
The SnapLogic GA build on the production server is also deployed on the UAT server.
UAT #2 Fixes
Platform
Fixes an issue where a Triggered Task invoked with a Groundplex URL fails to execute under the following two conditions:
The Snaplex contains a FeedMaster node.
The Do not start new execution if one is already active option in the Create Task dialog is selected.
Fixes an issue with a Scheduled Task where a Snaplex node is put into maintenance mode, and the Scheduled Task, which should be executed on a different node in the Snaplex, does not execute.
Fixes an issue where the expression toggle display does not render correctly for some users.
Fixes an issue where the Help hook for Pipeline execution statistics redirects to an incorrect documentation page.
Updates the Task monitoring display to make today's date default.
Fixes an issue where the Task Monitoring Dashboard does not display all metrics for a specified hour when seeking metrics for Tasks older than 24 hours.
Standard-mode Snaps
None
eXtreme-mode Snaps
None
4.21 Dot Releases
Platform
Date of Update
Snaplex Build
Updates
master-7588 - 4.21 Patch 3
This patch is applicable to only select customers and is not recommended.
master-7513 - 4.21 Patch 2 (Restricted)
This patch is applicable to only select customers and gives the option of reverting the ReplaceAll expression functionality to the 4.20 release. Contact Customer Support for more information.
master-7451 - 4.21 Patch 1 (Recommended)
Fixes an issue where if you invoke Triggered Tasks with only basic authentication without a bearer token, the IP address of your machine is exposed in the HTTP header.
Fixes an issue where the retry mechanism in an Ultra Task does not work when the input message is not acknowledged, hence removing the message from the FeedMaster queue.
Fixes an issue with the OAuth2 authentication across the platform where the authentication for endpoints intermittently fail because of timeouts.
Fixes an issue where a Pipeline does not complete an execution when invoked as a Triggered Task.
Improves the error messages that result from Pipelines not executing when in a queued state.
Fixes an issue where using HDFS ZipReader and HDFS ZipWriter Snaps as well as the Hadoop Directory Browser in Pipelines that run on Cloudplex instances result in a security exception error.
UI
Date of Update
Updates
Enhances Task and Snaplex error message responses.
Fixes an error with the eXtreme Accounts display, where you click Validate on the Create Account dialog, but the account creation fails even though the values of Access-Key ID and Secret Keys are correct.
Fixes an issue where the Pipeline field in the Pipeline Execute Snap with the New Form UI enabled does not accept user input.
Fixes an issue where refreshing Dashboard > Tasks causes the Start and EndDate selectors to change color.
Fixes an issue where the Pie Chart in the Data Preview Visualization display for an eXtreme Pipeline does not render undefined values as null, as in Standard mode.
Fixes an issue with the Designer Quick Tour where clicking Get Started > Next does not upload and display a Pipeline with the Salesforce Reader Snap.
Fixes an issue where the downloadable CSV file from Dashboard > Task subtab also includes details for the day after the date you specify.
Fixes the doc links for Activity Log, Pipeline Execution Statistics, Manager > Alerts, and Dashboard > Task displays.
Fixes the issue where Snowflake SCD2 Snap generates two output documents despite no changes to Cause-historizationfields with DATE, TIME and TIMESTAMP Snowflake data types, and with Ignore unchanged rows field selected.
Fixes the 'Connection pool shut down' error in S3 File Writer Snap by keeping the AmazonS3Client class object open until all the documents are processed.
Requires the use of a generic envelope while executing a create request through an NTLM account in the SOAP Execute Snap to ensure the output is rendered as expected.
Enhances the Kafka Consumer Snap with an option to allow reprocessing of records that remain unacknowledged due to acknowledge timeouts. This ensures that all records from Kafka are processed.
Enhances the JSON Generator Snap to include pass-through functionality where the Snap embeds the upstream input document under the original field of the output document along with other records.
Fixes the UnknownHostException error in Binary Snaps due to intermittent failures in FileSystemProvider registration, by replacing ServiceLoader with direct instantiations of FileSystemProvider class objects.
Fixes an error in Multi File Reader and File Delete Snaps during Pipeline execution where the Snaps fail to retrieve the temporary files from the given S3 location.
Introduces the productized version of SharePoint Online Snap Pack, which enables organizations to share and manage content and collaborate across an organization using the following SharePoint Online Snaps:
SharePoint Online - Search Items: Searches for the hierarchy of items matching the search query within a folder hierarchy, a whole drive, or files shared with the user.
Fixes the Directory Browser Snap that fails with an Unable to create filesystem object for smb:// error, by rolling back the SMB client library to the prior working version as in snapsmrc542
Fixes the Confluent Kafka Consumer Snap that aborts abruptly upon getting a CommitFailedException error. Upgraded the Kafka client library from version 2.2 to 2.5.
Fixes an issue in the Microsoft Dynamics AX Execute Snap that fails when using the Apache HTTPComponents library API if the request XML has Unicode characters.
Adds a new field, Handle Timestamp and Date Time Data, to Snowflake Lookup Snap. This field enables you to decide whether the Snap should translate UTC time to your local time and the format of the Date Time data.
Fixes the Binary Router Snap's failure in Ultra Pipelines by preventing a deep-copy (performing a copy of the original objects recursively) of binary data headers.
Fixes the issue in PGP Encrypt Snap where decrypting its output file using a third-party PGP software adds a random suffix and TMP extension in the file name. This issue occurs only when using the PGP Encrypt Snap in a Windows-based Groundplex.
Adds Hierarchical Data Format v5 (HDF5) support in AWS EMR. With this enhancement, you can read HDF5 files and parse them into JSON files for further data processing. See Enabling HDF5 Supportfor details.
Adds support for Python virtual environment to the PySpark Script Snap to enable reading HDF5 files in the S3 bucket.You can specifythe path for the virtual machine environment's ZIP file in this field.