Overview

Databricks is a data analytics platform optimized for the Microsoft Azure cloud services platform. It can also be run on Amazon AWS cloud and Google Cloud Platform. Databricks offers three environments for developing data-intensive applications: Databricks SQL, Databricks Data Science & Engineering, and Databricks Machine Learning. Learn more: Databricks Documentation.

Articles in this section

This Snap Pack focuses on the Databricks Data Science & Engineering environment which is also referred to as Databricks Lakehouse Platform (DLP) or Databricks. The Databricks Snap Pack contains the following Snaps:

Databricks - Select: Retrieves information from the target Databricks table.
Databricks - Insert: Inserts new rows of data in the target Databricks table.
Databricks - Delete: Deletes data from a target Databricks table.
Databricks - Bulk Load: Loads millions of rows of data in the target table through a single load operation.
Databricks - Unload: Unloads data from a target Databricks table through a single unload operation.
Databricks - Merge Into: Updates millions of existing rows and inserts new rows in a target Databricks table through a single operation.
Databricks - Execute: Runs multiple SQL statements on the target Databricks instance.

Limitations

With the basic authentication type for Databricks Lakehouse Platform (DLP) reaching its end of life on July 10, 2024, SnapLogic Databricks pipelines designed to use this authentication type to connect to DLP instances would cease to succeed. We recommend that you reconfigure the corresponding Snap accounts to use the Personal access tokens (PAT) authentication type.

Prerequisites

Create and configure an application corresponding to your Databricks instance in the AWS, Azure Portal, or Google Cloud Platform before using these Snaps. All the Snaps' accounts require information pertaining to this application for authentication purposes.

Supported Versions

This Snap Pack is tested against JDBC JAR version: databricks-jdbc-2.6.25-1.jar.

Using Alternate JDBC JAR File Versions

We recommend that you let the Snaps use this JAR file version. However, you may choose to use a different JAR file version.

Snap Pack History

Release	Snap Pack Version	Date	Type	Updates
August 2024	main27765	Aug 21, 2024	Stable	Upgraded the `org.json.json` library from v20090211 to v20240303, which is fully backward compatible.
May 2024	437patches27246	Aug 8, 2024	Latest	Added Databricks - Run Job. This Snap executes a job, checks its status in Databricks, and, based on the job's status, completes or fails the pipeline.
May 2024	437patches26400	May 15, 2024	Latest	Fixed an invalid session handle issue with the Databricks Snap Pack that intermittently triggered an error message when the Snaps failed to connect with Databricks to execute the SQL statement.
May 2024	main26341	May 8, 2024	Stable	Updated the Delete Condition (Truncates a Table if empty) field in the Databricks - Delete Snap to Delete condition (deletes all records from a table if left blank) to indicate that all entries will be deleted from the table when this field is blank, but no truncate operation is performed.
February 2024	main25112	Feb 14, 2024	Stable	Updated and certified against the current SnapLogic Platform release.
November 2023	main23721	Nov 8, 2023	Stable	Updated and certified against the current SnapLogic Platform release.
August 2023	main22460	Aug 16, 2023	Stable	Updated and certified against the current SnapLogic Platform release.
May 2023	433patches21630	Jun 28, 2023	Latest	Enhanced the performance of the Databricks - Insert Snap to improve the amount of time it takes for validation.
May 2023	main21015	May 10, 2023	Stable	Upgraded with the latest SnapLogic Platform release.
February 2023	main19844	Feb 9, 2023	Stable	Upgraded with the latest SnapLogic Platform release.
November 2022	main18944	Nov 10, 2022	Stable	The Databricks - Insert Snap now creates the target table only from the table metadata of the second input view when the following conditions are met: The Create table if not present checkbox is selected. The target table does not exist. The table metadata is provided in the second input view.
September 2022	430patches18305	Sep 29, 2022	Latest	The name of the Databricks - Multi Execute Snap is simplified to Databricks - Execute Snap. The Use Result Query checkbox in the Databricks - Execute Snap enables you to include in the Snap's output the result of running (during validation) each SQL statement specified in the Snap. The Retry mechanism for the Databricks Snap Pack enables the following Databricks Snaps to repeatedly perform the selected operations for the specified number of times when the Snap account connection fails or times out. Databricks - Delete Databricks - Insert Databricks - Select Databricks - Execute Databricks - Bulk Load (when the Source Type is Input View) Databricks - Merge Into (when the Source Type is Input View) The following fields are added to each Databricks Snap as part of this enhancement: Number of Retries: The number of attempts the Snap should make to perform the selected operation when the Snap account connection fails or times out. Retry Interval (seconds): The time interval in seconds between two consecutive retry attempts.
September 2022	430patches17796	Sep 28, 2022	Latest	The Manage Queued Queries property in the Databricks Snap Pack enables you to decide whether a given Snap should continue or cancel executing the queued Databricks SQL queries.
August 2022	main17386	Aug 11, 2022	Stable	Upgraded with the latest SnapLogic Platform release.
4.29.2.0	42920rc17045	Jul 15, 2022	Latest	A new Snap Pack for Databricks Lakehouse Platform (Databricks or DLP) introduces the following Snaps: Databricks - Select: Retrieves information from the target Databricks table. Databricks - Insert: Inserts new rows of data in the target Databricks table. Databricks - Delete: Deletes data from a target Databricks table. Databricks - Bulk Load: Loads millions of rows of data in the target table through a single load operation. Databricks - Unload: Unloads data from a target Databricks table through a single unload operation. Databricks - Merge Into: Updates millions of existing rows and inserts new rows in a target Databricks table through a single operation. Databricks - Multi Execute: Runs multiple SQL statements on the target Databricks instance.

SnapLogic Documentation

Databricks Snap Pack

Overview

Limitations

Prerequisites

Supported Versions

Snap Pack History

Related Links