Page Comparison

...

If the last Snap in the Pipeline takes 2 to 5 seconds to update the runtime, the ELT Pipeline statistics are not displayed even after the Pipeline is completed. The UI does not auto-refresh to display the statistics after the runtime.
Workaround: Close the Pipeline statistics window and reopen it to see the ELT Pipeline statistics.
When you return to the Snap Statistics tab from the Extra Details tab in the Pipeline Execution Statistics pane, it contains the status bar (Pipeline execution status) instead of the Download Query Details hyperlink and the individual counts of Records Added, Records Updated, and Records Deleted.
When your Databricks Lakehouse Platform instance uses Databricks Runtime Version 8.4 or lower, ELT operations involving large amounts of data might fail due to the smaller memory capacity of 536870912 bytes (512MB) allocated by default. This issue does not occur if you are using Databricks Runtime Version 9.0.
ELT Pipelines created prior to 4.24 GA release using one or more of the ELT Insert-Select, ELT Merge Into, ELT Load, and ELT Execute Snaps may fail to show expected preview data due to a common change made across the Snap Pack for the 4.26 GA release. In such a scenario, replace the Snap in your Pipeline with the same Snap from the Asset Palette and configure the Snap's Settings again.
In case you are writing into a Snowflake target table, this Snap attempts to create the target table even when it exists in the database.
Suggestions displayed for the Schema Name field in this Snap are from all databases that the Snap account user can access, instead of the specific database selected in the Snap's account or Settings.

...

Info

title	SQL Functions and Expressions for ELT

You can use the SQL Expressions and Functions supported for ELT to define your Snap or Account settings with the Expression symbol = enabled, where available. This list is common to all target CDWs supported. You can also use other expressions/functions that your target CDW supports.

...

Parameter Name Data Type Description Default Value Example

Label

String

Insert excerpt

	File Writer
	File Writer
nopanel	true

ELT Insert-Select

Insert Employee Records

Get preview data

Checkbox

Multiexcerpt include macro

name	getpreviewdata
page	ELT Intersect

Not selected

Selected

Database Name

String

Required. Enter the name of the database in which the target table is located. Leave it blank to use the database name specified in the account settings.

If your target database is Databricks Lakehouse Platform (DLP), you can, alternatively, mention the file format type for your table path in this field. For example, DELTA, CSV, JSON, ORC, AVRO. See Table Path Management for DLP section below to understand the Snap's behavior towards table paths.

N/A

TESTDB

Schema Name (Not applicable to Databricks Lakehouse Platform)

String

Required. Enter the name of the database schema. In case it is not defined, then the suggestion for the schema name retrieves all schema names in the specified database when you click Image Modified.

Multiexcerpt macro

name	ME_Schema_Name

Ensure that you include the exactly same schema name including the double quotes, if used, when you repeat the schema name in the Target Table Name field.
Leave this field blank if your target database is Databricks Lakehouse Platform.

N/A

"TEST_DATA"

Target Table Name

String

Required. The name of the table or view into which you want to insert the data.

Info
Only views that can be updated (have new rows) are listed as suggestions. So, Join views are not included. This also implies that the Snap account user has the Insert privileges on the views listed as suggestions.

If your target database is Databricks Lakehouse Platform (DLP), you can, alternatively, mention the target table path in this field. Enclose the DBFS table path between two `(backtick/backquote) characters. For example, `/mnt/elt/mytabletarget`. Learn more about the Snap’s behavior toward paths in theTable Path Management for DLP section.

Note
If you choose to include the schema name as part of the target table name, ensure that it is the same as specified in the Schema Name field, including the double quotes. For example, if the schema name specified is "table_schema_1", then the table name should be "table_schema_1"."tablename".

Note

If the target table or view does not exist during run-time, the Snap creates one with the name that you specify in this field and writes the data into it. During Pipeline validation, the Snap creates the new table or view but does not write any records into it.

The new table or view will not be dropped if a subsequent/downstream Snap failure occurs during validation.
Use double quotes (““) to specify the table or view name if you want to include special characters such as hyphens (-) in the table or view name.
A table or view name must always start with a letter.
Integers and underscores (_) can also be a part of the name.
All characters are automatically converted to uppercase at the backend. Use double-quotes to retainlowercase.

Multiexcerpt macro

name	ME_Schema_And_Table_Names

Ensure that you include the exactly same schema name, if at all, including the double quotes as specified in the Schema Name field.

N/A

"TEST_DATA"."DIRECT"

EMPLOYEE_DATA

EMPLOYEE_123_DATA

REVENUE"-"OUTLET

"net_revenue"

Advanced Options Checkbox Select this checkbox to define the mapping of your source and target table columns scenario with or without the Insert Expressions list. When selected, it activates the Operation Types field. Deselected Selected

Operation Types

Dropdown list

Choose one of the following options that best describes your source data and the INSERT preference:

Source Columns Order. Use this operation type to fill the target table columns with the source table data in the same order of columns as in the source table.
Some Source and Target Column names are identical. Use this operation type when you want only a subset of the target table columns that are identical to the source to be filled with the source data and optionally define the Insert Expression List to fill the remaining columns in the target table.
All Source and Target Column names are identical. Use this operation type when your source and target table have the same set of column names. You cannot specify any Insert Expressions with this operation type.

Source Columns Order

Some Source and Target Column names are identical

Table Option List

This field set enables you to specify the table options you want to use on the target table. These options are populated based on the Snap Account (target CDW) selected. You must specify each table option in a separate row. Click to add rows.

This field set contains one field:

Table Option

Table Option

String/Suggestion

Click the Suggestions icon ( ) to view and select the table option you want to apply for loading data into the target table.

N/A

DISTRIBUTION = HASH ( cust_name )

Insert Expression List

This field set enables you to specify the values for a subset of the columns in the target table. The remaining columns are assigned null values automatically. You must specify each column in a separate row. Click to add rows.

This field set is disabled if you select All Source and Target Column names are identical in the Operation Types field.

This field set consists of the following fields:

Insert Column
Insert Value

Note
You can use this field set to insert data only into an existing table.

Insert Column String Enter the name of the column in the target table to assign values. N/A ORD_AMOUNT

Insert Value

String

Enter the value to assign in the specified column. Repeat the column name if you want to use the values in the source table. You can also use expressions to transform the values.

N/A

ORD_AMOUNT

ORD_AMOUNT+20

Overwrite Checkbox Select to overwrite the data in the target table. If not selected, the incoming data is appended. Not selected Selected

...

Versions Compared

Old Version 54

New Version 55

Key