...

...

• The Manage Queued Queries property in the Databricks Snap Pack enables you to decide whether a given Snap should continue or cancel executing the queued Databricks SQL queries.

• The Retry mechanism for the Databricks Snap Pack enables the following Databricks Snaps to repeatedly perform the selected operations for the specified number of times when the Snap account connection fails or times out.

  • Databricks - Delete

  • Databricks - Insert

  • Databricks - Select

  • Databricks - Bulk Load (when the Source Type is Input View)

  • Databricks - Merge Into (when the Source Type is Input View)

The following fields are added to each Databricks Snap as part of this enhancement:
Number of Retries: The number of attempts the Snap should make to perform the selected operation when the Snap account connection fails or times out.
Retry Interval (seconds): The time interval in seconds between two consecutive retry attempts.

Known Issues

Databricks Snap Pack: Due to an issue with DLP, aborting a Databricks Pipeline validation (with preview data enabled) causes only those SQL statements that retrieve data using bind parameters to get aborted while all other static statements (that use values instead of bind parameters) persist. For example, select * from a_table where id = 10 will not be aborted while select * from test where id = ? gets aborted.
To avoid this issue, ensure that you always configure your Snap settings to use bind parameters inside its SQL queries.

...

• Due to an issue with the Redshift COPY INTO SQL for PARQUET input files, the ELT Load Snap fails to perform ALTER TABLE actions before loading data from a PARQUET source file to the target table in Redshift. This issue does not occur when the source file is in CSV or another format.

• Due to an issue with BigQuery table schema management (the time travel feature), an ALTER TABLE action (Add or Update column) that you attempt after deleting a column (DROP action) in your BigQuery target table causes the table to break and the Snap to fail.

  • As a workaround, you can consider either avoiding ALTER TABLE actions on your BigQuery instance using the Snap or creating (CREATE) a temporary copy of your table and deleting (DROP) it after you use it.

• Due to an issue with Databricks Runtime Version 11 and above, the ELT Aggregate Snap fails to calculate the value for the linear regression aggregate function REGR_R2 for the target DLP instance and returns a cast exception. As a workaround, you can revert your Databricks Runtime Version to 10.5 or below.

• ELT Pipelines built using an ELT SCD2 or an ELT Load Snap fail to perform the DROP AND CREATE TABLE and ALTER TABLE operations on Delta tables when working with a Databricks SQL persona on the AWS Cloud with the error Operation not allowed: ALTER TABLE RENAME TO is not allowed for managed Delta tables on S3. However, the same action runs successfully with the Data Science and Engineering persona on the AWS Cloud.

  • This is due to the limitation with the Databricks SQL Admin Console that does not allow adding the configuration parameter spark.databricks.delta.alterTable.rename.enabledOnAWS true to its SQL Warehouse Settings.

• Due to an issue with the Redshift COPY INTO SQL for PARQUET input files, the ELT Load Snap fails to perform ALTER TABLE actions before loading data from a PARQUET source file to the target table in Redshift. This issue does not occur when the source file is in CSV or another format.

• The ELT Load Snap cannot automatically infer schema from GZIP-compressed source files located in an ADLS Gen2 storage location for loading the data into Snowflake target tables.

• Because BigQuery does not support Automatic Schema Inferencing (ASI) from source files in Amazon S3, the ELT Load Snap fails to produce the required output when you choose Drop and Create table as the Load Action and define the Target to Source Column Map in the Snap’s settings. However, the same configuration works fine for Pipeline validation purposes.

• Snowflake does not support COPY statements that result in data transformation in any of the validation modes it allows. Therefore, the ELT Load Snap fails during Pipeline validation when the number of columns you define in the Target to Source Column Map field set is not the same as the number of columns in the target table. But the same Pipeline runs successfully because this limitation does not apply to Pipeline execution.

  • As a workaround for this limitation, ensure that you specify error_on_column_count_mismatch=false in the Format Option field under the File Format Option List field set.

• While you specify an SQL statement in the SQL Statement Editor of an ELT Snap as an expression, the dynamic validation for the expression displays inline errors when there is more than one incoming document and without the '__sql__' key to the current Snap, when you select Get Preview Data checkbox in the previous Snap, and when Preview Document Count in your user settings is set to a value more than 1.

  • To prevent this error and similar ones, do not select the Get Preview Data checkbox in the previous Snap, set the Preview Document Count in your user settings to 1, or append a condition where 1 = 0 to the SQL statement with the Get Preview Data checkbox selected.

• Due to a limitation in Snowflake in inferring the source files (and thereby the source schema) based on file name patterns in a Storage Integration-based session, the ELT Load Snap fails to load data into your Snowflake target instance based on any File Name Pattern that you specify. As a workaround, we recommend that you use the File List field set to define the source data file locations for the Snap to load the data.

• If you are using Azure Active Directory password--based authentication on the older Snap Pack version, you could encounter an error while using ELT Pipelines in Azure Synapse. This is due to absence of the following two JAR files in the ELT Snap Pack. Add these two JAR files to the list that you want to use to resolve the issue:

  • MSAL4J (Microsoft Authentication Library for Java) during compile time.

  • Nimbusds oauth for runtime.

  However, this error would not occur when you use the latest Snap Pack version (version number will included once released) 430patches18196.

IIP UI Redesigns

The UI for SnapLogic IIP is redesigned for the September release.

...