/
ELT Minus

ELT Minus

May 06, 2024

In this article

An account for the Snap

You must define an account for this Snap to communicate with your target CDW. Click the account specific to your target CDW below for more information:

Overview

Use this Snap to add a MINUS SQL operator to the separate queries coming from upstream Snaps. The MINUS SQL operator corresponds to ANSI EXCEPT. In Azure Synapse and BigQuery, this Snap uses the EXCEPT operation unlike in Redshift and Snowflake, where the Snap uses the MINUS operation. This Snap also allows you to preview the result of the MINUS SQL operation on the incoming SQL queries. You can validate the modified query using this preview functionality.

  • The MINUS SQL operation does not eliminate duplicate records. You can add the ELT Unique Snap to the ELT Minus Snap to remove duplicates. 

  • This Snap's output is not commutative. The Snap's output changes based on which table is connected in which input view. For example, if you use queries for tables A and B as inputs, where query A is connected to the first input and query is connected to the second, the result from the Snap's execution would not be the same if it was the other way around. Basically, A MINUS B is not the same as B MINUS A.

Prerequisites

None.

Limitation

ELT Snap Pack  does not support Legacy SQL dialect of Google BigQuery. We recommend that you use only the BigQuery's Standard SQL dialect in this Snap.

Known Issues

In any of the supported target databases, this Snap does not appropriately identify nor render column references beginning with an _ (underscore) inside SQL queries/statements that use the following constructs and contexts (the Snap works as expected in all other scenarios):

  • WHERE clause (ELT Filter Snap)

  • WHEN clause

  • ON condition (ELT Join, ELT Merge Into Snaps)

  • HAVING clause

  • QUALIFY clause

  • Insert expressions (column names and values in ELT Insert Select, ELT Load, and ELT Merge Into Snaps)

  • Update expressions list (column names and values in ELT Merge Into Snap)

  • Secondary AND condition

  • Inside SQL query editor (ELT Select and ELT Execute Snaps)

Workaround

As a workaround while using these SQL query constructs, you can:

  • Precede this Snap with an ELT Transform Snap to re-map the '_' column references to suitable column names (that do not begin with an _ ) and reference the new column names in the next Snap, as needed.

  • In case of Databricks Lakehouse Platform where CSV files do not have a header (column names), a simple query like SELECT * FROM CSV.`/mnt/csv1.csv` returns default names such as _c0, _c1, _c2 for the columns which this Snap cannot interpret. To avoid this scenario, you can:

    • Write the data in the CSV file to a DLP table beforehand, as in: CREATE TABLE csvdatatable (a1 int, b1 int,…) USING CSV `/mnt/csv1.csv` where a1, b1, and so on are the new column names.

    • Then, read the data from this new table (with column names a1, b1, and so on) using a simple SELECT statement.

  • In case of Databricks Lakehouse Platform, all ELT Snaps' preview data (during validation) contains a value with precision higher than that of the actual floating point value (float data type) stored in the Delta. For example, 24.123404659344 instead of 24.1234. However, the Snap reflects the exact values during Pipeline executions.

Snap Input and Output

Input/Output

Type of View

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input/Output

Type of View

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 2

  • Max: No limit

  • ELT Select

  • ELT Aggregate

Multiple SQL queries.

Output

Document

  • Min: 1

  • Max: 1

  • ELT Unique

  • ELT Insert-Select

The incoming SQL queries with the MINUS operator included. 

Snap Settings

SQL Functions and Expressions for ELT

You can use the SQL Expressions and Functions supported for ELT to define your Snap or Account settings with the Expression symbol = enabled, where available. This list is common to all target CDWs supported. You can also use other expressions/functions that your target CDW supports.

Parameter Name

Data Type

Description

Default Value

Example 

Parameter Name

Data Type

Description

Default Value

Example 

Label

String

ELT Minus

Exclude Common Records

Get preview data

Checkbox

Select this checkbox to include a preview of the query's output. The Snap performs limited execution and generates a data preview during Pipeline validation.

In the case of ELT Pipelines, only the SQL query flows through the Snaps but not the actual source data. Hence, the preview data for a Snap is the result of executing the SQL query that the Snap has generated in the Pipeline.

The number of records displayed in the preview (upon validation) is the smaller of the following:

  • Number of records available upon execution of the SQL query generated by the Snap.

  • The value set in the Preview Data Count setting (default is 50 records).

Rendering Complex Data Types in Databricks Lakehouse Platform

Based on the data types of the fields in the input schema, the Snap renders the complex data types like map and struct as object data type and array as an array data type. It renders all other incoming data types as-is except for the values in binary fields are displayed as a base64 encoded string and as string data type.

In Azure Synapse and BigQuery, this Snap uses EXCEPT operation unlike in Redshift and Snowflake, wherein the Snap uses MINUS operation.

Output preview in Azure Synapse

Not selected

Selected

Retain duplicates (Databricks Lakehouse Platform only)

Checkbox

If your target database is a Databricks Lakehouse Platform instance, select this checkbox to indicate that the Snap can include duplicate entries in its output. Else (by default), the Snap retrieves only unique records from the MINUS/EXCEPT operation result.

Not selected

Selected

Troubleshooting

None.

Examples

Retrieving Uncommon Rows From a Table

We need a query with the MINUS operator if we want to compare two tables and retrieve the uncommon records from the first table. This example shows how we can use the ELT Minus Snap to build this query.

First, we build SELECT queries to read the target tables. To do so, we can use two ELT Select Snaps, in this example: Read Part A and Read Part B. Each of these Snaps is configured to output a SELECT * query to read the target table in the database. Additionally, these Snaps are also configured to show a preview of the SELECT query's execution as shown:

Read Part A Configuration

Read Part B Configuration

Read Part A Configuration

Read Part B Configuration

A preview of the outputs from the ELT Select Snaps is shown below:

Read Part A Output

Read Part B Output

Read Part A Output

Read Part B Output

Then, we connect the ELT Minus Snap to the output view of the ELT Select Snaps. The SELECT * queries in both of these Snaps form the inputs for the ELT Minus Snap. Upon execution, the ELT Minus Snap combines both incoming SELECT * queries and adds the MINUS operator.

A preview of the ELT Minus Snap's output is shown below:

We can also add an ELT Insert-Select Snap downstream and write the result of this query into another table.

.

Downloads

Important Steps to Successfully Reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.

  2. Configure Snap accounts as applicable.

  3. Provide Pipeline parameters as applicable.



See Also