Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
absoluteUrltrue
excludeOlder Versions|Additional Resources|Related Links|Related Information

Overview

Snap type:

Write


Description:

This Snap executes a Snowflake bulk upsert. The Snap bulk updates the records if present , or inserts records to the target Snowflake table. Incoming documents are first written to a staging file on Snowflake's internal staging area. A temporary table is created on Snowflake with the contents of the staging file. An update operation is then run to update existing records in the target table and/or an insert operation is run to insert new records into the target table.

Depending on the account that you configure with the Snap, you can use the Snowflake Bulk Upsert Snap to upsert data into AWS S3 buckets or Microsoft Azure Storage Blobs.

Note

The Snap creates temporary files in JCC when the staging location is internal and the data source is Input view. These temporary files are removed automatically once the Pipeline completes execution.


Prerequisites:

Insert excerpt
Snowflake - Bulk Load
Snowflake - Bulk Load
nopaneltrue

Security prerequisites: You should have the following permission in your Snowflake account to execute this Snap:

  • Usage (DB and Schema): Privilege to use the database, role, and schema.

The following commands enable minimum privileges in the Snowflake Console:

Code Block
grant usage on database <database_name> to role <role_name>;
grant usage on schema <database_name>.<schema_name>;

For more information on Snowflake privileges, refer to Access Control Privileges.

The below are mandatory when using an external staging location:

When using an Amazon S3 bucket for storage:

  • The Snowflake account should contain S3 Access-key ID, S3 Secret key, S3 Bucket and S3 Folder.
  • The Amazon S3 bucket where the Snowflake will write the output files must reside in the same region as your cluster.

When using a Microsoft Azure storage blob:

  • A working Snowflake Azure database account.
Internal SQL Commands

This Snap uses the following commands internally:

  • SELECT FROM TABLE: SELECT as a statement enables you to query the database and retrieve a set of rows. SELECT as a clause enables you to define the set of columns returned by a query.

  • COPY INTO <table>: Enables loading data from staged files to an existing table. 

  • CREATE TEMPORARY TABLE: Enables creating temporary table in the database. See Create Temporary Table Privilege for more information.

  • MERGE: Enables inserting, updating, and deleting values in a table based on values in a second table or a subquery.

Support and limitations:
  • Works in Ultra Task Pipelines.
  • The special character '~' is not supported if it is there in temp directory name for Windows. It is reserved for user's home directory.
Account: 

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. The S3 BucketS3 Access-key ID, and S3 Secret key properties are not needed for the Snowflake Bulk Upsert Snap as Snowflake uses its internal staging area to host the input data. See Configuring Snowflake Accounts for information on setting up this type of account.

Views:


InputThis Snap has exactly one document input view. 
OutputThis Snap has at most one document output view. 
ErrorThis Snap has at most one document error view and produces zero or more documents in the view. 


Settings

Label


Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Schema name 

The database schema name. Selecting a schema filters the Table name list to show only those tables within the selected schema.

Note

The values can be passed using the pipeline parameters but not the upstream parameter.


Table name


Required. Table on The table on which to execute the bulk load operation.

Note

The values can be passed using the pipeline parameters but not the upstream parameter.


Staging location

The type of staging location that is used for data loading.

The options available include:

  • External: Location that is not managed by Snowflake. The location should be an AWS S3 Bucket or Microsoft Azure Storage Blob. These credentials are mandatory while validating the Account.
  • Internal: Location that is managed by Snowflake.

Default value:  Internal

File format object

Specifies an existing file format object to use for loading data into the table. The specified file format object determines the format type such as CSV, JSON, XML, and AVRO, or other format options for data files.

Default value:  [None]

File format typeSpecifies a predefined file format object to use for loading data into the table. The available file formats include CSV, JSON, XML and AVRO.
Default value: NONE
File format option

Specifies the file format option; separate multiple options by using blank spaces and commas.

Info

You can use various file format options including binary format which passes through in the same way as other file formats. See File Format Type Options for more information.

Before loading binary data into Snowflake, you must specify the binary encoding format, so that the Snap can decode the string type to binary types before loading into Snowflake. This can be done by specifying the following binary file format:

BINARY_FORMAT=xxx (Where XXX = HEX|BASE64|UTF-8)

However, the file you upload and download must be in similar formats. For instance, if you load a file in HEX binary format, you should specify the HEX format for download as well.

Default value:  [None]

Encryption type

Specifies the type of encryption to be used on the data. The available encryption options are:

  • None: Files do not get encrypted.
  • Server Side Encryption: The output files on Amazon S3 are encrypted with the server-side encryption.
  • Server-Side KMS Encryption: The output files on Amazon S3 are encrypted with Amazon S3-generated KMS key. 

Default value: No default value.

Note

The KMS Encryption option is available only for S3 Accounts (not for Azure Accounts) with Snowflake.


Note

If Staging Location is set to Internal, and when Data source is Input view, the Server Side Encryption and Server-Side KMS Encryption options are not supported for Snowflake snaps:

This happens because Snowflake encrypts loading data in its internal staging area and does not allow the user to specify the type of encryption in the PUT API (see Snowflake PUT Command Documentation.)


KMS key

The KMS key that you want to use for S3 encryption. For more information about the KMS key, see AWS KMS Overview and Using Server Side Encryption.

Default value: No default value.

Note

This property applies only when you select Server-Side KMS Encryption in the Encryption Type field above.


Buffer size (MB)

The size of each part, or chunk, of data being uploaded, in MB. 

Minimum value: 6 MB

Maximum value: 5000 MB

Default value: 10 MB

The Snap uses the S3 Multipart Upload API to write files to S3. For more details, see Amazon S3 Multipart Upload Limits.

Key columns


Required. Columns to use to check for existing entries in the target table.

Default:  None 
 

Delete Upsert Condition


Delete Upsert Condition when true, causes the case to be executed.
Default:  None


Preserve case sensitivity
Insert excerpt
MySQL - Insert
MySQL - Insert
nopaneltrue


Manage Queued Queries


Select this property to decide whether the Snap should continue or cancel the execution of the queued Snowflake Execute SQL queries when you stop the pipeline.

Note

If you select Cancel queued queries when the pipeline is stopped or if it fails, then the read queries under execution are cancelledcanceled, whereas the write type of queries under execution are not cancelledcanceledSnowflake internally determines which queries are safe to be cancelled canceled and cancels those queries.

Default value: Continue to execute queued queries when the pipeline is stopped or if it fails

Load empty strings

If selected, empty string values in the input documents are loaded as empty strings to the string-type fields. Otherwise, empty string values in the input documents are loaded as null. Null values are loaded as null regardless.

Default value:  Selected

Multiexcerpt include macro
nameSnap Execution
pageSOAP Execute

Multiexcerpt include macro
nameExecution_Detail_Write
pageSOAP Execute

Example

Upserting Records

This example Pipeline demonstrates how you can efficiently update and delete data (rows) using the Key Column ID field and Upsert Delete condition. We use Snowflake - Bulk Upsert Snap to accomplish this task.
Image Modified

First, we configure the Mapper Snap with the required details to pass them as inputs to the downstream Snap.
Image Modified

After validation, the Mapper Snap prepares the output as shown below to pass to the Snowflake Bulk - Upsert Snap.
Image Modified

Next, we configure the Snowflake - Bulk Upsert Snap to:

  • Upsert the existing row for P_ID column, (so, we provide P_ID in the Key column field).
  • Delete the rows where the FIRSTNAME is snaplogic in the target table, (so, we specify FIRSTNAME = 'snaplogic' in the Delete Upsert Condition field).

After execution, this Snap inserts a new record into the existing row for the P_ID key column in Snowflake.

Inserted Records Output in JSONInserted Records in Snowflake

Upon execution, if the Delete Upsert condition is true, the Snap deletes the records in the target table as shown below.

Output in JSONDeleted Record in Snowflake

Download this Pipeline.

Bulk Loading Records

In the following example, we update a record using the Snowflake Bulk Upsert Snap. The invalid records which cannot be inserted will be routed to an error view. 

In the pipeline execution:

  1. The Mapper (Data) Snap maps the record details to the input fields of Snowflake Upsert Snap:
  2. Snowflake Upsert Snap updates the record using the "PRASANNA"."Persons" table object:



  3. Successful execution of the pipeline displays the following data preview (Note that one input has been made and two records with the name P_ID have been updated within the Persons table.):

Attachments
patterns.*slp, .*zip

Insert excerpt
Snowflake Snap Pack
Snowflake Snap Pack
nopaneltrue