August 2021, 4.26 Release Notes

In this article

Key New Features in SnapLogic

Platform:

  • Zero Downtime during quarterly release updates.
  • Snaplex-based Scheduler will be the default scheduling mechanism for all Scheduled Tasks after the auto-upgrade on September 18.
  • Ability to encrypt SnapLogic user emails.

API Management: 

  • Ability to manage lifecycle of APIs.
  • Redesigned API Catalog and API documentation for the Developer Portal.
  • Support for creating APIs from existing SnapLogic Assets.

New Snaps:

  • Shopify Snap Pack.
  • Tableau Snaps—Hyper Formatter, Hyper Parser, and Hyper Write.
  • Zuora Snaps—Zuora OpenAPI and Zuora REST Read.

ELT for Cloud Data Platforms:

  • Support for Databricks Lakehouse Platform as a target database.
  • Ability to run multiple SQL queries using ELT Execute Snap.

New! Introducing SnapLogic Flows


4.26 UAT Delta

 Click here to expand...

This section tracks the changes made during the iterative pushes to the UAT server and the GA release. The following is the release schedule:

  • UAT 1: Friday, July 30, 2021 – Snaplex version: 4.26_rc-10116 - 4.26 GA (Recommended)

  • UAT 2: Friday, August 06, 2021 

  • GA: Saturday, August 14, 2021 (9 p.m. PT)  

The SnapLogic GA release build on the production server is also deployed on the UAT server.

Important

  • Per the SnapLogic Release Process, all remaining Snaplex instances across Orgs are auto-upgraded to the main-10245 - Patch 1 version at 9 p.m. PT on September 18, 2021. Orgs migrated to the 4.26 GA version prior to the auto-upgrade are not impacted. 

    The Snaplex upgrade process also upgrades the Java version to the latest version. After the upgrade, the Java version of your Cloudplex nodes may not be the same as your FeedMaster nodes. However, this does not impact the operations of your SnapLogic instance. We will be fixing this issue in a subsequent release.

  • To use the new SnapLogic features and Snaps in the 4.26 release, ensure that your Snaplex nodes are upgraded to the main-10245 - Patch 1 (Recommended) version.
  • This 4.26 release includes the patches deployed to SnapLogic after the 4.25 release, as listed in the 4.25 Dot Releases section. 

4.26 Dot Releases

Platform

Date of UpdateSnaplex BuildUpdates
 main-10631 - Patch 3
  • Fixed an issue that caused Pipelines to be stuck in the NoUpdate state.
  • Fixed an issue where some validated Pipelines would not show preview data.
  • Fixed an issue where some schedule tasks would fail to execute.
  • Fixed an issue that prevented the Snowflake Snap from loading JAR files.
main-10451 - Patch 2
  • Fixed an issue where the FeedMaster node would not enter maintenance mode. 
  • Fixed an issue where the Load Balancer node would return an error instead of stopping further requests when in maintenance mode.
N/A

Control Plane update:

  • Fixed an issue where system administrators would not receive password reset emails if email encryption was enabled. 
  • Fixed an issue where users were unable to see Tasks under the Task Column of the Dashboard
  • Fixed an issue where opening Ultra Task Details would return an Invalid ID error.
  • Fixed an issue where Ultra Tasks would continuously restart.
  • Fixed an issue that prevented users from updating a Snaplex with the Manage Snaplex Versions setting if the Snaplex had Slack users or channels added.
  • Fixed an issue that prevented users from creating APIs in projects that had Tasks with empty Snaplex instances configured. 
  • Added a read-only view for non-admin users to the Developer Portal.
 main-10245 - Patch 1 (Recommended)
  • Fixed an issue where Scheduled Tasks would execute twice after switching between control plane scheduling and Snaplex scheduling.
  • Fixed an issue where a Snaplex would wait an hour before starting to execute scheduled jobs after switching from control plane level scheduling to Snaplex level scheduling.

UI

Date of UpdateSnaplex BuildUpdates
 N/A
  • Fixed an issue where the Join Snap would fail after its input view was updated.
  • Fixed an issue where the default value of newly-added properties would not appear in existing instances of the Snaplex.
  • Fixed an issue where the Router Snap returned an error if a view was renamed multiple times.
  • Fixed an issue where some Pipeline Execute Snaps were missing the Reuse executions to process documents checkbox.
  • Fixed an issue that prevented the conditional fields of the Team Operations Snap and SQS Account from rendering.

Standard

Snap PackDate of UpdateSnaplex BuildUpdates
REST426patches12749

Fixed an issue with the REST Post Snap, where the Snap delayed the fetching of documents, when the batch size is greater than or equal to one.

SQL Server426patches12535

Fixed an issue in the SQL Server Bulk Load snap where boolean data type failed to load into BIT field.

Binary
Hadoop
426patches12288

Fixed memory leak issue when using the HDFS protocol in Hadoop and Binary Snaps.

Transform

426patches11922Fixed an issue with the Join Snap, where it exhausted the memory while buffering millions of objects.
Binary426patches11922Fixed an issue with the File Writer Snap, where the Snap failed to open a channel while accessing through a SFTP protocol.
Salesforce426patches12054Enhanced the Salesforce Subscriber Snap with improved resilience to network failures and fixed an issue with duplicate documents.
REST426patches11746

Enhanced the REST Post Snap to display all cookies (securedauth cookies and httponly cookies) from the CookieStore along with headers when you select the Show all headers checkbox.

Transform426patches11780Fixed an issue in the XML Formatter Snap, where the Map input to first repeating element in XSD checkbox is selected, while no XSD is specified for mapping the input.
RabbitMQ426patches11770

Enhanced the RabbitMQ Account with a new checkbox Cluster Support that enables multiple cluster node support for RabbitMQ Consumer and RabbitMQ Producer Snaps. You can select this checkbox to enable high availability of clusters by adding multiple hosts (Host name and Port) for each cluster node.

Marketo 426patches11555Fixed an issue with the Marketo Lead Read Snap, wherein the Snap generated the same next page token anytime during the day.
Email 426patches 11591

Fixed the following issues with the Email Archive Snap where the Snap:

  • Failed with an error User is authenticated but not connected, as the Snap did not re-authenticate with the server after the connection expired due to the long durations in the execution of Ultra Pipelines. 
  • Caused multiple copies of the same source message in the target folder when the Snap was executed multiple times in preview.
Transform 426patches11725Fixed an issue with the Join Snap wherein the upstream document flow of the right view is blocked by the left view, which hung the Join Snap.
DynamoDB426patches11593Fixed an issue with the DynamoDB Update Snap, where the data type conversion for input keys caused an error.
Binary 426patches11529
  • Fixed an issue with the Decompress Snap, where the Snap prematurely uncompressed multi-stream BZIP2 files.
  • Enhanced the PGP Encrypt Snap with improved memory usage and also fixed an issue with this Snap, where the Snap failed to encrypt large size files due to the lack of memory.
Workday 426patches11525Enhanced the Workday Read Snap with the new field Pool Size that controls the maximum number of threads in the pool. This field is available only when you select the Multi-threaded checkbox.
Snowflake 426patches11469Fixed an issue with Snowflake Insert and Snowflake Bulk Load Snaps where the schema names or database names containing an underscore (_) caused the Pipelines to time out.
Active Directory 426patches11280Fixed an issue with the Active Directory Add Member Snap that failed when using forward slash "/" in the Distinguished name field.
SharePoint Online main89Enhanced all Snaps in the SharePoint Online Snap Pack to support dynamic site and document library values using input document variables.

Data Automation

Snap PackDate of UpdateSnap Pack Build NameUpdates
ELT 426patches12534
  • Fixed an issue with ELT Transform Snap where it may display incorrect schema only in the previews (during Pipeline validation). This occurs especially when the incoming SQL statement (defined in the SQL Statement Editor of the upstream Snap) contains one or more of the WHERE, GROUP BY, HAVING , ORDER BY, LIMIT, LIMIT followed by OFFSET, and SAMPLE clauses. Here are a few Pipeline scenarios where this issue might surface:

ELT 426patches12021
  • Fixed an issue where the ELT Load Snap connecting to a Databricks Lakehouse Platform (DLP) instance failed to perform the load operation. Ensure that you provide a valid DBFS Folder path in the Snap's account settings as the Snap requires this folder path.
ELT 426patches11646
  • Enhanced the ELT Database Account to support token-based authentication (Source Location Session Credentials) to S3 locations for Snowflake and Redshift target databases.
  • Enhanced the ELT Aggregate Snap with the following changes:
    • Revised the field labels from:
      • GROUP BY Fields List field set > Output Field to GROUP BY Field.
      • ORDER-By Fields to ORDER-BY Fields (Aggregate Concatenation Functions Only).
    • Removed the Suggestion option for Field Name field under General Aggregate Functions List field.
    • Made the Alias Name fields in the Aggregate Concatenation Functions List and the Percentile Distribution Functions List field sets mandatory.
  • If your target database is a Databricks Lakehouse Platform (DLP) instance, then the ELT Load Snap supports loading data from source CSV files that contain only comma as the separator between values.
ELT 426patches11323
  • Enhanced the ELT Database Account to allow parameterization of field values using Pipeline Parameters. You can define and use these parameters in expression-enabled fields to pass values during runtime.

ELT 426patches11262
  • Fixed the following Known Issues recorded in the 4.26 GA version:
    • For a Snowflake target instance, the ELT Insert Select Snap does not suggest column names to select for the Insert Column field in the Insert Expression List.
    • The Snaps—ELT Merge Into, ELT Select, ELT Join, and ELT Filter—do not prevent the risk of SQL injection when your target database is Databricks Lakehouse Platform (DLP).
    • Intermittent null-pointer exceptions in the ELT Load Snap on Databricks Lakehouse Platform (DLP).

    • The ELT Insert Select Snap attempts to create the target table even when it exists in the Snowflake database.
    • When loading data from a JSON file into a target Databricks Lakehouse Platform (DLP) instance using an ELT Load Snap, if you choose the Drop and Create Table option as the Load Action and specify an additional column (that is not available in the JSON file) for the new table, it results in one more column null added to the new target table.
    • When you use the SQL editor in the ELT Select Snap configuration to define your SQL query, the Pipeline validation fails due to a syntax error in the following scenarios. However, the Pipeline execution works as expected. The only workaround is to drop the LIMIT clause and the optional OFFSET clause from the SQL query during Pipeline validation.
      • The query contains a LIMIT clause on a Snowflake, Redshift or Databricks Lakehouse Platform target instance: The SQL query created during Pipeline validation includes an additional LIMIT clause, for example: SELECT * FROM "STORE_DATA"."ORDERS" LIMIT 10 LIMIT 990

      • The query contains an OFFSET clause (supported in case of Snowflake and Redshift): The SQL query created during Pipeline validation looks like SELECT * FROM "STORE_DATA"."ORDERS" LIMIT 10 OFFSET 4 LIMIT 990

Platform 

Snaplex-based Scheduler

During the 4.26 release cycle's auto-upgrade date of September 18, the Snaplex-based Scheduler will be enabled by default through the Control Plane. The new scheduling mechanism improves the timeliness and reliability of Scheduled Task executions. You do not need to update your Snaplex instances to enable the new scheduler, unless you have set the feature flag to false for your Org, in which case you will have to switch it to true in order for the Snaplex-based Scheduler to be enabled for your Org. Contact support@snaplogic.com for more information. 

Zero downtime of the SnapLogic Platform during the release windowStarting with this release, we are implementing zero downtime for the SnapLogic Platform during quarterly release updates. Thus, you will be able to access the SnapLogic UI and all your Pipelines will run as schedule during the release window. For details, see the SnapLogic release process article.

Email Encryption: Adds the ability to upload a public key in Manager to be used for emails. You can now encrypt your emails using this public key. Email Encryption is a subscription feature.

New APIM Features

Lifecycle Management for APIs: Added the capability to manage the lifecycle of your SnapLogic APIs. You can now publish APIs created from API specifications to the new developer portal in the Portal Manager console. You can also unpublish, deprecate, and retire an API version, providing full API lifecycle management features.

Portal Manager: Added console for managing the developer portal for your Org. You can now customize the new Developer Portal with your own branding and URL suffix. The Portal Manager also enables you to view API status on the API Catalog.

API Developer Portal: Introduced a new page where API consumers can explore and view APIs created in your Org. The API Catalog provides a space where APIs can be exposed to users outside of the SnapLogic ecosystem. You control the consumer's ability to view and call APIs by adding API policies in the API Version Details page.

Create APIs from Assets in Manager: Added options in the API Manager console to create APIs directly from Projects and the Assets. You can now build your APIs by developing Pipelines and Tasks, and add other assets like Accounts and Files to the Project to be uploaded as an API in the API Manager. You can also create an Empty API asset as a placeholder.

OAuth 2.0 Credentials API Policy: Added a new OAuth2 flow to authenticate users using client credentials in the API Policy.

Enhancements to Existing Features

Batching Support: Added batching support to the Pipeline Execute Snap. In contrast to the Reuse mode, the batching field enables users to specify the number of documents to be processed to completion through the child Pipelines before processing the next document in the batch. Accordingly, batch mode does not support reuse mode.

Windows 2019 Support: Added Windows 2019 support for your Groundplex instances.

SnapLogic will sunset support for Windows Server 2012 on . Hence, ensure that you upgrade your Groundplex instances to Windows 2016 or 2019. 

Snaplex Tab Dashboard Improvements: Snaplex tab widgets now display one-minute data intervals for time ranges up to seven days.

OAuth Accounts: Optimized the OAuth account refresh operations by moving them to the customer Groundplex instances. Additionally, the Box OAuth 2.0 account type was also added to Snaps that do not restart Ultra Task instances because Box has the capability of reloading the new account token.

Known Issues

  • Few Snaplex instances with an unusually high number of tasks may take slightly longer to start scheduling tasks when a node is starting.
  • For Scheduled Tasks scheduled every 5–10 minutes, the next execution time is not being updated even though the Task is executed.
  • In the Version Details page of an API, clicking Edit Details from the API > Versions page displays the Server field for selecting another Snaplex, but the field is disabled if the specified Snaplex is in a down state.

Behavior Change

  • Users are no longer unsubscribed from a Snap Pack if the Snap Pack's name is changed.
  • Your Groundplex thread/heap dump files are now stored in a folder as configured in the java.io.tmpdir property, which is usually /tmp on Linux.


Standard-mode Snaps

Behavior Fix in the CSV Formatter Snap

In 4.26, we fixed the Ignore empty stream checkbox functionality in the Snap to actually produce an empty binary stream output in case there is no input document when the checkbox is NOT enabled. Previously, there was no output even when the checkbox was NOT enabled.

If your existing Pipelines (prior to 4.26) use the Ignore empty stream functionality, then you might need to revisit your Pipeline design and either enable or disable the said checkbox depending upon the expected behavior.

New Snaps

Binary Copy: This Snap that belongs to the Flow Snap Pack enables you to copy a binary stream to the Snap’s output views. Use this Snap if you want to send the same information to multiple endpoints.

Copybook Snap Pack: This Snap Pack enables you to work with Cobol Copybooks. This Snap Pack has the following Snaps:

  • Cobol Copybook Data Generation: Generates sample records based on Cobol Copybooks.
  • Cobol Copybook Parser: Parses Cobol Copybook and data.

Shopify: This Snap Pack allows businesses to set up an online store and sell their products online. You can use this Snap Pack to create orders, products, customers, and run automated workflows. The Shopify Snap Pack has the following Snaps:


Tableau

  • The Snap Pack consisting of the Tableau 8 version Snaps (Tableau 8 Write and TDE 8 Formatter) is deprecated due to no customer usage. See the Deprecated Snaps section for more information.
  • Starting from the 4.26 release, the existing Tableau 9 & 10 Snap Pack is renamed to Tableau Snap Pack.

Added the following Tableau Snaps to support hyper extract files for Tableau 10.25 and later versions. Hyper is Tableau's in-memory data engine that is optimized for fast data ingestion and analytical query processing on large or complex data sets.


Zuora: Added the following Snaps and account types to connect these Snaps with the Zuora REST API. 

Updated Snap Packs

Amazon SQS

  • Enhanced the Amazon SQS Account with a new checkbox Enable Large Message Support that enables Amazon SQS client to send and receive messages up to 2GB through Amazon S3. 
  • Enhanced the AWS S3 and Amazon SQS accounts to add the Snap’s information to the User Agent header for Cross IAM account requests.

AnaplanAdded the following fields to the Anaplan Write Snap:

  • Override server settings
  • Separator
  • Text Delimiter

Azure Active Directory: Enhanced the Create Entry and Update Entry Snaps to support Pipeline parameters and upstream values for the Attribute name setting under Attributes and the Snap Pack to support proxy authentication. 

Binary

  • Enhanced the Directory Browser Snap to process files with the owner field for the SFTP protocol. Upon validation, the output of the Snap fetches the owner’s name, which was previously displayed as unknown.
  • Fixed an issue with the File Writer Snap where the Snap fails to connect to the SFTP endpoint using the AWS Transfer Family.
  • Fixed an issue where Pipelines fail to connect the SFTP Server with the Binary Snap Pack after upgrading to the SnapLogic 4.25 release version (main9554).
  • Removed the Project ID field from the Google Service Account, since the JSON Key already contains information about the Project ID. 

Box:

  • Modified default values of the following fields across the Snap Pack to trigger automatic retries by default, in case of connection failures or timeouts:
    • Number of Retries - Changed from 0 to 3
    • Retry Interval (seconds) -  Changed from 1 to 5

Google SheetsEnhanced the Worksheet Writer Snap to populate the target schema preview with headers and associated data types (when the data is written to an existing worksheet with a valid header) in the upstream Snap.

JDBC:

  • Fixed an issue with Generic JDBC - Select Snap where if there are URL properties in the account, the unPooledConnection does not use the properties when creating a connection. This causes the auto-detect functionality to use the wrong database name.
  • Fixed an issue with MySQL and MariaDB in Generic JDBC - Update Snap when a table is incorrectly reported as not being found.

Kafka:

  • Enhanced the Kafka Producer Snap to support the Avro serialization format with the new target data type Avro under Key and Value Serializer.
  • Enhanced the Kafka SSL Account with new fields (Registry Username or Key and Registry Password or Secret) for Schema Registry authentication. The two existing SASL properties (SASL Username and SASL Password) are revised to SASL Username or Key and SASL Password or Secret, respectively.

  • Enhanced the Kafka Producer and Kafka Consumer Snaps to create a default client ID string that identifies the specific Snap instance.
  • Improved the handling of interrupted/aborted Kafka Snaps to ensure a proper clean-up of metrics.

  • Optimized the Kafka Producer Snap to initialize the Kafka API only if there is at least one input document.

  • Fixed an issue of account passwords being included in the log messages output of Kafka Snaps. The account passwords are now hidden in the logs for both Kafka Consumer and Kafka Producer Snaps.

  • Apache Kafka client library is upgraded from version 2.6.0 to 2.8.0.

  • Confluent Kafka client libraries are upgraded from version 5.2.1 to 6.2.0.

MySQL:

  • Enhanced the MySQL - Lookup Snap suggestions to load the Output Fields and Lookup Column settings.
  • Fixed an issue with the MySQL - Select Snap where the second output view of the Snap was showing empty column values and improved error messages in the Snap where the Snap fails with a null pointer exception error when the given account information is invalid.

OpenAPIEnhanced the OpenAPI Snap with the following two fields:

  • HTTP Headers
  • Query Parameters

PostgreSQL: Enhanced the performance of PostgreSQL - Bulk Load Snap significantly. We expect the Snap to execute up to three times faster than the previous version for enterprise workloads.

REST

  • Added Send Client Data as Basic Auth Header checkbox to the Create Account settings of the OAuth2 and OAuth2 SSL accounts to include and pass the client credentials information in the header when connecting to the OAuth endpoint.
  • Added a new account type REST AWS SigV4 that adds the ability to call any AWS API.

Salesforce: Enhanced the Salesforce Read Snap to enable you to add an optional second output view to display the schema of a target object as the output document. 

ServiceNowEnhanced the ServiceNow QueryServiceNow InsertServiceNow Update, and ServiceNow Delete Snaps with a retry mechanism that includes the following fields:

  • Number Of Retries
  • Retry Interval (seconds)

Snowflake:

  • Enhanced the Snowflake S3 Database and Snowflake S3 Dynamic accounts with S3 AWS Token field that allows you to connect to private and protected Amazon S3 buckets.
  • Added a new account type Snowflake Google Storage Database to connect to Google Cloud Storage to load data.
  • Added support for all existing Snowflake Snap accounts to connect to a Snowflake instance hosted on the Google Cloud Platform

  • Enhanced Snowflake - Lookup and Snowflake SCD2 Snaps with the Input Date Format field to select from the following two options:
    • Continue to execute the snap with the given input Date format
    • Auto Convert the format to Snowflake default format

SOAPEnhanced the SOAP Execute Snap with a new checkbox Escape special characters to escape XML special characters in variable values when inserting values into the Velocity template.

SQL Server: Fixed an issue with the SQL Server - Bulk Load Snap where the Snap fails when the login password contains a colon or a less than (<) symbol.

Transform:

  • Enhanced the CSV Parser Snap with a new checkbox Preserve Surrounding Spaces that enable the Snap to preserve the surrounding spaces for the values that are non-quoted.
  • Enhanced the JSON Splitter Snap with a new field Show Null Values for Include Paths that enables the Snap to show key-value entries of the null values for the objects added to the Include Paths field in the output document.
  • Enhanced the Join Snap with a new field Available Memory Threshold (%) to enable the Snap to keep all the Right input view documents with the same join-path values in memory until the join operation is done for the specific join-path values. 
  • Replaced Strict XSD output field in the XML Formatter Snap with Map input to first repeating element in XSD. If selected, the Snap ignores the root element from the XSD file.  

Improved the error messages in Teradata Snap Pack, Oracle Snap Pack, MySQL - Select in MySQL Snap Pack, Channel Operations Snap in Slack Snap Pack, Create Event Snap in Exchange Online and Teams - Create Team in the Teams Snap Pack where the Snaps fail with a null pointer exception error when the given account information is invalid. 

Enhanced the JIRACoupa, SOAP Snap Packs and Workday Prism Analytics Bulk Load Snap in Workday Prism Analytics Snap Pack to support HTTP proxy authentication.

Updated the AWS SDK from version 1.11.688 to 1.11.1010 in the DynamoDB Snap Pac