Fall 2017 - 4.11 Release Notes

In this Page

Before You Begin

You may need to clear your browser cache before you log into the latest SnapLogic Elastic Integration Platform.

See the appropriate documentation for your browser:

You may see an Upgrade Notice dialog informing you that you will be logged out immediately to complete the update process. Because the SnapLogic Elastic Integration Platform consists of multiple applications, you will likely see this message multiple times.

Accounts and Platform Updates

Some accounts may have a fixed time for refresh tokens, such as Google accounts that must be refreshed every hour. If that refresh needs to occur when the platform is down for an update, the refresh does not occur. To prevent these accounts from failing after a new platform deployment, it is recommended that you refresh your accounts before the designated down time. If the platform is down for longer than an hour, you will need to refresh those accounts once the platform is back online.

Browsers

It is recommended that you upgrade to the latest version of your browser. 

Platform

Enhancements

General

  • Migration of encrypted account information is supported across trusted organizations.
  • Expression language now supports the spread operator.
  • Optimizations in the streaming of binary data across Snaps.
  • Enhancements in error handing during pipeline preview and execution.
  • Additional metadata for Tasks when using the Metadata Read Snaps.
  • When users change a password, they receive an email notification. For service accounts, the notification is sent to the user creator of the account.
  • Groundplex can be configured to communicate to the control plane through a reverse HTTP /wiki/spaces/SD/pages/1439269.
  • Enhancements in enforcement of expression language syntax to increase conformity.

Designer and Manager

  • You can now compare pipelines across projects. You are no longer restricted to comparing pipelines within a project or project space.
  • You can create a unique error pipeline and add it as a child pipeline to other pipelines in your project. 
  • New accounts you create in SnapLogic are recognized when you migrate projects from one org to another.
  • Project and project space owners can set project permissions based on user groups. Previously, owners had to set permissions on a user-by-user basis.
  • As you create expressions, they are validated dynamically for syntax and spelling errors and to provide output previews.
  • To prevent buffer overflow failures, you can set /wiki/spaces/SD/pages/1437907 to control the Kryo serializer buffer size.
  • When Snaps are set to stop execution, you can now preview error documents. You don't need to open the error view to see them.  

Spark SQL

SnapLogic Read

  • Read Task output includes task history.

Public APIs

Snaps 

This section provides the Snap updates for the Fall 2017 (4.11) release. This release also includes the patches made to Snaps in the 4.10.0 release, as listed in the 4.10 Snap Patch Releases page. For information on Snap Pack build versions, see Snap Pack Versions by Release (updated at product release).

Pipelines which are already started before a Snap update will continue to run with the older version of the Snaps. Only pipelines started after the new Snaps are deployed will run with the new Snaps.

For ultra pipelines, the currently running instances would continue to run with the older Snap packs. Editing and saving the task instance in Manager will cause a rolling restart of the ultra pipeline instances and the new Snap Packs will get picked up. Only those customers using SQL Server with Windows authentication and SAP will need to restart their Groundplex instances.


Some accounts may have a fixed time for refresh tokens, for example Google and Box accounts must be refreshed every hour. If that refresh needs to occur when the platform is down for an update, the refresh does not occur. To prevent these accounts from failing after a new platform deployment, it is recommended that you refresh your accounts before the designated down time.

New Snap Packs

Microsoft Dynamics AX: The Snap Pack includes the following Snaps: Dynamics AX Create, Dynamics AX Read, Dynamics AX Delete, Dynamics AX Execute, Dynamics AX Update and Dynamics AX Search.

Updated Snap Packs

Binary: 

Cassandra: Where clause property in Cassandra Select updated to support values containing upstream parameters and Pipeline parameters.

Confluent Kafka: Updated the Confluent Kafka SSL Account with SCRAM_SASL support.

DynamoDB Query: Added a new DynamoDB Query Snap with Secondary Index support.

Google BigQuery:

Hadoop: Added Kerberos and zone encryption support to the Standard mode Parquet Reader and /wiki/spaces/SD/pages/1438227 Snaps.

MySQL: 

Redshift: Added SSL support to the Configuring Redshift Accounts.

REST: Updated support of NTLM Account Authentication to the REST Snap Pack. 

Salesforce: Updated the Polling timeout property's default value to 3000 for Salesforce Read, Salesforce Poller, Salesforce Lookup, and Salesforce SOQL Snaps.

SAP HANA: Completed testing Support Package Stack 12 (STS 12).

Script: Added a new PySpark Snap to submit spark-submit jobs from a Groundplex edge node.

ServiceNow: Completed testing support of ServiceNow Istanbul version.

Spark: Added a new Spark mode Snap SparkSQL Execute.

Spark Script: Deprecated the /wiki/spaces/SD/pages/1437917 Snap from this release. Do migrate to the PySpark Snap.

Snowflake: Updated Snowflake Execute Snap with a new Use Result Query property to write the query results to the output view. 

SOAP: Updated support of NTLM Account Authentication to the SOAP Snap Pack.

Transform:

  • Updated the JSON Formatter Snap with Binary Header and Content properties to allow the Snap to pass through binary header data.
  • Enhanced the In-memory Lookup Snap for Performance Optimization.
  • Enhanced the Sort Snap for Performance Optimization.

Known Issues

  • When Resource Manager kills the Spark pipeline application, then the status displays as "running," while it is actually failed. 
  • The SPARK_HOME environment variable should not be set when starting a Groundplex. If it is set, the Groundplex fails to start. To work around this issue:
    1. Open a command shell and enter the following command to remove the environment variable:
      $ unset SPARK_HOME
    2. Restart the JCC process.
  • If you are using a Local Dashboard, it can take up to 10 minutes after the JCCs start (or restart) before users can log in.
  • Spark pipelines fail to execute on a Hadooplex, when running on Kerberos-enabled Hadoop Cluster and user impersonation enabled, by throwing a Java Exception.
  • Some Snap doc links are failing to launch the respective documents from the SnapLogic UI. You can either search for them within the SnapLogic documentation portal or use the links in the New Snap Packs section above. 

UAT Delta

UAT Delta

This section tracks the changes made during the iterative pushes to the UAT server and between the last push to UAT and the GA release. The expected schedule is:

  • UAT Push #1, October 24, 2017
  • UAT Push #2, October 28, 2017
  • UAT Push #3, November 4, 2017
  • GA Push, November 11, 2017

UAT #2 Delta

Snap Updates

  • Snowflake Bulk Load: Fixed an issue that occurred when creating new tables for staged files.
  • Redshift SCD2: Fixed an issue with input validation.
  • RabbitMQ Producer and Consumer: Fixed an issue with specifying input values for exchange keys, exchange names, and queue name properties.
  • Dynamics CRM: Added support for GUID data types to pass an input parameter to Dynamics CRM Snaps.
  • Parquet Reader: Fixed an issue with omitting columns that include null values.
  • Join: Removed conflicting dependencies from the Snap pack dependency management.
  • RabbitMQ Producer and Consumer in Ultra Mode: Fixed an issue with retry logic when the pipeline fails in a negative scenario.

Platform Updates

  • Enabled Pipeline properties (like pipe.label) in Expression libraries and error pipeline parameters.
  • Fixed an expression language issue with escaped quote characters.
  • Fixed permission issue when looking up pipeline runtimes by ID.
  • Fixed networking issues communicating with Groundplex nodes.

UI Updates

  • Fixed issues with Dynamic Validation messages appearing when they were distracting (or not relevant).
  • Fixed an issue where Snap execution time that appeared on the Dashboard was incorrect.
  • Deleting and adding Snaps caused Dynamic Validation errors. This issue is fixed. 

Big Data Updates

  • Fixed the code merge conflict issue which caused validations to fail.
  • Fixed Spark context getting locked when multiple validations were run simultaneously.
  • Fixed issue with standard mode HDFS reader/writer after the delegation token expired.

UAT #3 Delta

Snap Updates

  • Google BigQuery Execute: Added support for byte data type.
  • Google BigQuery Write: Added support for Array and Record data types.
  • AzureSQL Bulk Load: Fixed an issue with Snap timeout for Completed/Cancelled pipelines.
  • Confluent Kafka Consumer: Fixed an issue to ensure the auto-commit interval is respected, even when there are no new incoming messages to process.
  • Confluent Kafka: Shortened the default Account validation timeout value to 15 seconds and enhanced the timeout error message.
  • Redshift Bulk Load: Changed the data encoding algorithm to enhance performance and to more efficiently utilize memory.
  • Hadoop Snaps: Fixed an issue with Suggest functionality in the HDFS Reader/Writer and Parquet Reader/Writer Snaps' Settings that utilize values provided as expressions.
  • Excel Parser: Fixed an issue with the XLS file and parsing columns when the values are null at the end of the row.
  • SAP Execute: Fixed an issue where pipelines could continue to reuse invalid cached credentials after Snap Account values are modified.
  • Workday Snaps: Added additional logging and introduced request/response compression to improve Snap performance.
  • Snowflake Bulk Load: Fixed an issue with parsing of the AWS S3 file URLs when the file is located at the root of the S3 bucket.

Platform Updates

  • Fixed an SSO login-related deployment issue.
  • Fixed an issue with Task type during project migration.
  • Fixed an issue that caused the pipeline prepare process to time out when Snap download was slow.
  • Added support for larger file sizes when saving preview data.
  • Fixed an issue with Snaplex Count computation for Insights data.

UI Updates

  • Fixed an issue where required fields on the Account tab are empty, but appear to be validated. 
  • Fixed several issues where incorrect, "Cannot read property" script errors appeared.
  • Removed an unnecessary error message that appeared when Snaps were deleted and the user logs back in. 

Big Data Updates

  • Fixed the memory leak in JCC that happened after executing Spark pipelines.
  • If a cluster is overloaded, the Spark driver is terminated.
  • Fixed execution of Spark pipelines on Kerberized clusters.
  • Fixed a Spark context locked issue that occurred in Spark mode pipeline validation.
  • Fixed a file descriptor (FD) leak that occured during Spark validation.
  • Fixed an issue with all validations showing as "Failed" after one validation fails. 

GA Delta

Snap Updates

  • Confluent Kafka Snap Pack: Fixed an issue where Account validation and listing topics would fail due to ACL.
  • SAP Execute Snap: Addressed an issue when a list of primitives is provided as input for a TABLE_TYPE data type.
  • Snowflake Snap Pack: Added support for Snap settings' values for buckets and folder, overriding the values set in the Account.
  • SparkSQL Snap Pack: Minor update of default Snap labels, including CSV Parser, HDFS Reader, HDFS Writer, and Parquet Parser. Note that existing pipelines with the older Snap names will continue to execute successfully. 
  • Google BigQuery: Reset default SQL Dialect to Legacy SQL from Standard SQL.

Platform Updates

  • Fixed an issue with empty expressions when running with an older version of the expression library.

UI Updates

  • Fixed several issues where incorrect, "Cannot read property" script errors appeared when saving or validating a pipeline.

Big Data Updates

  • Added log for yarn resource utilization on each cluster node for cases when the JCC terminates the driver.
  • Fixed an issue where the Spark application was terminated when a Snap Pack took more than 4 minutes to download.