ELT Union

Owned by Mohammed Iqbal
Last updated: Mar 22, 2024 by Gouri Bhagchandani
2 min read

In this article

An account for the Snap

You must define an account for this Snap to communicate with your target CDW. Click the account specific to your target CDW below for more information:

Overview

Use this Snap to combine the incoming SQL queries and add a UNION ALL or UNION DISTINCT SQL operator between them. The UNION ALL SQL operator is used to combine the result of two or more SQL SELECT statements. The UNION DISTINCT SQL operator does the same and removes duplicate rows.

This Snap also allows you to preview the result of the output query. You can validate the modified query using this preview functionality.

Prerequisites

  • Two or more SQL SELECT queries.
  • The source tables in the SQL SELECT statements must have the same number of columns.
  • The column data types of each of the source tables also need to match.

Limitation

ELT Snap Pack  does not support Legacy SQL dialect of Google BigQuery. We recommend that you use only the BigQuery's Standard SQL dialect in this Snap.

Known Issues

In any of the supported target databases, this Snap does not appropriately identify nor render column references beginning with an _ (underscore) inside SQL queries/statements that use the following constructs and contexts (the Snap works as expected in all other scenarios):

  • WHERE clause (ELT Filter Snap)
  • WHEN clause
  • ON condition (ELT Join, ELT Merge Into Snaps)
  • HAVING clause
  • QUALIFY clause
  • Insert expressions (column names and values in ELT Insert Select, ELT Load, and ELT Merge Into Snaps)
  • Update expressions list (column names and values in ELT Merge Into Snap)
  • Secondary AND condition

  • Inside SQL query editor (ELT Select and ELT Execute Snaps)

Workaround

As a workaround while using these SQL query constructs, you can:

  • Precede this Snap with an ELT Transform Snap to re-map the '_' column references to suitable column names (that do not begin with an _ ) and reference the new column names in the next Snap, as needed.
  • In case of Databricks Lakehouse Platform where CSV files do not have a header (column names), a simple query like SELECT * FROM CSV.`/mnt/csv1.csv` returns default names such as _c0, _c1, _c2 for the columns which this Snap cannot interpret. To avoid this scenario, you can:
    • Write the data in the CSV file to a DLP table beforehand, as in: CREATE TABLE csvdatatable (a1 int, b1 int,…) USING CSV `/mnt/csv1.csv` where a1, b1, and so on are the new column names.
    • Then, read the data from this new table (with column names a1, b1, and so on) using a simple SELECT statement.
  • In case of Databricks Lakehouse Platform, all ELT Snaps' preview data (during validation) contains a value with precision higher than that of the actual floating point value (float data type) stored in the Delta. For example, 24.123404659344 instead of 24.1234. However, the Snap reflects the exact values during Pipeline executions.

Snap Input and Output

Input/OutputType of ViewNumber of ViewsExamples of Upstream and Downstream SnapsDescription
Input 

Document

  • Min: 2
  • Max: No limit
  • ELT Select
  • ELT Intersect
  • ELT Copy
The SQL SELECT statements that you want to combine with a UNION DISTINCT or UNION ALL operator.
Output

Document

  • Min: 1
  • Max: 1
  • ELT Insert-Select
  • ELT Limit

The modified SQL query with a UNION DISTINCT or UNION ALL operator.