Snowflake - Execute

In this article

Overview

You can use this Snap to execute arbitrary Snowflake SQL. This Snap works only with single queries.

The Snap substitutes the valid JSON paths that are defined in the WHERE clause for queries/statements with the values present in the incoming document. If the incoming document does not carry a substituting value, the document is written to the error view. If the Snap executes a SELECT query, it merges the query results into the incoming document and overwrites the values of all existing keys. On the other hand, the Snap writes the original document if there are no results from the query. If an output view is available and an UPDATE/INSERT/MERGE/DELETE statement is executed, the original document that was used to create the statement becomes output with the status of the statement executed.

Snap Type

The Snowflake - Execute Snap is a Write-type Snap that executes arbitrary Snowflake SQL. Depending on your query type, this Snap can also read data.

Prerequisites

Security Prerequisites

You should have the following permissions (but not limited to) in your Snowflake account to execute this Snap:

  • Usage (DB and Schema): Privilege to use database, role, and schema.

  • Create table: Privilege to create a table on the database. role, and schema.

The following commands enable minimum privileges in the Snowflake Console:

grant usage on database <database_name> to role <role_name>; grant usage on schema <database_name>.<schema_name>; grant "CREATE TABLE" on database <database_name> to role <role_name>; grant "CREATE TABLE" on schema <database_name>.<schema_name>;

For more information on Snowflake privileges, refer to Access Control Privileges.

Internal SQL Commands

The permissions to grant for usage on database and creating tables depend on the queries you provide in this Snap.

Support for Ultra Pipelines

Works in Ultra Pipelines. 

Limitations

  • User-defined functions (UDFs) created in the Snowflake console can be executed using Snowflake - Execute Snap. See Execute-Snowflake Execute Snap supports UDFs in examples. 

  • The Snap may break existing Pipelines if the JDBC Driver is updated to a newer version.

Snowflake Execute and Multi-Execute Snaps may break existing Pipelines if the JDBC Driver is updated to a newer version.

With the updated JDBC driver (version 3.12.3), the Snowflake Execute and Multi-Execute Snaps' output displays a Status of "-1" instead of "0" without the Message field upon successfully executing DDL statements. If your Pipelines use these Snaps and downstream Snaps use the Status field's value from these, you must modify the downstream Snaps to proceed on a status value of -1 instead of 0.

This change in the Snap behavior follows from the change introduced in the Snowflake JDBC driver in version 3.8.1:
"Statement.getUpdateCount() and PreparedStatement.getUpdateCount() return the number of rows updated by DML statements. For all other types of statements, including queries, they return -1."

Known Issues

Because of performance issues, all Snowflake Snaps now ignore the Cancel queued queries when pipeline is stopped or if it fails option for Manage Queued Queries, even when selected. Snaps behave as though the default Continue to execute queued queries when the Pipeline is stopped or if it fails option were selected.

Behavior Change

  • In 4.26, when the stored procedures were called using the Database Execute Snaps, the queries were treated as write queries instead of read queries. So the output displayed message and status keys after executing the stored procedure.
    In 4.27, all the Database Execute Snaps run stored procedures correctly, that is, the queries are treated as read queries. The output now displays message key, and OUT params of the procedure (if any). The status key is not displayed.

  • If the stored procedure has no OUT parameters, only the message key is displayed with value success.

  • From 4.30 Release, the Snowflake Execute Snap writes the output value as-is for FLOAT or DOUBLE datatype columns if these columns have the value as NaN (Not a Number). Earlier, the Snap displayed an exception error when the FLOAT or DOUBLE datatype column has the value as NaN. This behavior is not backward compatible.

If you have any existing Pipelines that are mapped with status key or previous description then those Pipelines will fail. So, you might need to revisit your Pipeline design.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input

Document

  • Min: 0

  • Max: 1

  • JSON Generator

  • Binary to Document

Incoming documents are first written to a staging file on Snowflake's internal staging area. A temporary table is created on Snowflake with the contents of the staging file. An update operation is then run to update existing records in the target table and/or an insert operation is run to insert new records into the target table.

Output

Document

  • Min: 0

  • Max: 1

  • Mapper

  • Snowflake Execute

If an output view is available, then the output document displays the number of input records and the status of the bulk upload as follows:

 

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

Field Name

Field Type

Description

Field Name

Field Type

Description

Label*

 

Default Value: Snowflake - Execute
Example: Load Employee Tables

 

String

Specify the name for the Snap. You can make the name more specific, especially if your Pipeline has more than one of the same Snap.

SQL Statement*

 

Default Value: N/A

Example: INSERT into SnapLogic.book (id, book) VALUES ($id,$book)

 

String/Expression

Specify the Snowflake SQL statement to execute on the server.

Document value substitution is performed on literals starting with '$', for example, $people.name is substituted with its value available in the incoming document.

In DB Execute Snaps, if the Snowflake SQL statement is not an expression, the JSON path, such as $para, is allowed in the WHERE clause only.

If the query statement starts with SELECT (case-insensitive), the Snap regards it as a select-type query and executes once per input document. If not, the Snap regards it as a write-type query and executes in batch mode.

This Snap does not allow you to inject Snowflake SQL, for example, select * from people where $columnName = abc.
Only values can be substituted since it uses prepared statements for execution, which, for example, results in select * from people where address = ?.

Without using expressions

  • EmpId = 12 

  • email = 'you@example.com'

Using expressions

  • "EMPNO=$EMPNO and ENAME=$EMPNAME"

  • email = $email 

  • emp=$emp

  • "emp='" + $emp + "'"

  • "EMPNO=" + $EMPNO + " and ENAME='" + $EMPNAME+ "'"

Query type

 

Default Value: Auto
Example: Read

Dropdown list/Expression

Select the type of query for your SQL statement (Read or Write).

When Auto is selected, the Snap tries to determine the query type automatically.
If the execution result of the query is not as expected, you can change the query type to Read or Write.

Pass through

 

Default Value: Selected

Checkbox

Select this checkbox to enable the Snap to pass the input document to the output view under the key named original. This option applies only to the Execute Snaps with SELECT statement.

Ignore empty result

 

Default Value: Deselected

Checkbox

Select this checkbox to not write any document to the output view when a SELECT operation does not produce any result. If this checkbox is not selected and the Pass-through checkbox is selected, the input document is passed through to the output view.

Number of Retries

 

Default Value: 0
Example: 3

Integer

Specify the maximum number of retry attempts the Snap must make in case there is a network failure and is unable to read the target file. The request is terminated if the attempts do not result in a response.

  • If the Number of retries value is set to 0 (the default value), the retry option is disabled, and the Snap does not initiate a retry. The pipeline will not attempt to retry the operation in case of a failure—any failure encountered during the database operation will immediately result in the pipeline failing without any retry attempts to recover from the errors.

  • If the Number of retries value is greater than 0, the Snap initiates a download of the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and then attempts to download the file again from the beginning. After the download is successful, the Snap streams the data from the temporary file to the downstream pipeline. All temporary local files are deleted when they are no longer needed.

Minimum value: 0

Retry Interval (seconds)

 

Default Value: 1
Example: 10

Integer

Specify the time interval between two successive retry requests. A retry happens only when the previous attempt resulted in an exception. 

Use Result Query

 

Default Value: Deselected

Checkbox

Select this checkbox to write the query execution result to the Snap's output view after the successful execution. The output of the Snap will be enclosed within the key Result Query, and the value will be the actual output produced by the SQL query. See the example Snowflake Execute with Use Result Query enabled to know more about this option. 

This option allows users to effectively track the query's execution by clearly indicating the successful execution and the number of records affected, if any, after the execution.

Handle Timestamp and Date Time Data

 

Default value: Default Date Time format in UTC Time Zone

Example: SnapLogic Date Time format in Regional Time Zone

Dropdown list

Choose an option for handling timestamp and date time data. The available options are:

  • Default Date Time format in UTC Time Zone: The Snowflake date time data are represented in UTC Time Zone.

  • SnapLogic Date Time format in Regional Time Zone: The Snowflake date time data are represented in the same regional time zone value, as provided in the Snowflake account.

Manage Queued Queries

 

Default value: Continue to execute queued queries when pipeline is stopped or if it fails

Example: Cancel queued queries when the Pipeline is stopped or if it fails

Dropdown list

Select an option from the list to determine whether the Snap should continue or cancel the execution of the queued Snowflake Execute SQL queries when you stop the Pipeline. The available options are:

  • Continue to execute queued queries when pipeline is stopped or if it fails

  • Cancel queued queries when pipeline is stopped or if it fails

 

Default Value: Execute only
Example: Validate & Execute

Dropdown list

Examples 

Snowflake Execute with Use Result Query enabled

This example Pipeline demonstrates how to insert data into a table using the Snowflake Execute Snap.

First, we configure the Snowflake Execute Snap as follows. Note that we select the Use Result Query checkbox to view the statement result output.



Upon execution, we see the following output enclosed within the key Result Query.

The following screenshot displays the output preview when we disable the Use Result Query checkbox.




Executing the Snowflake SQL query using the Execute Snap


The following example demonstrates the execution of Snowflake SQL query using the Snowflake Execute Snap.

First, we configure the Execute Snap with this query—select * from "PRASANNA"."ADOBEDATA" , which returns the data from ADOBEDATA. 

Upon successful execution, the Snap displays the following output in its data preview.

Snowflake Execute Snap supports UDFs

User-defined functions (UDFs) created in the Snowflake console can be executed using Snowflake - Execute Snap. In the following example, the SQL statement is defined and then the Snap is executed with that conditions. 

First, the Snowflake Execute Snap is used to give the user-defined SQL statement. area_of_circle(3.0) is a UDF here. The Snap settings and the output view are as follows:

Then the Mapper Snap is used to define columns that need to be picked up from the Output of the Snowflake Execute. 

See Also

https://docs.snowflake.com/en/developer-guide/udf/sql/udf-sql-scalar-functions.html

https://docs.snowflake.com/en/sql-reference/udf-overview.html

https://docs.snowflake.com/en/user-guide-getting-started.html

Snap Pack History