Input

Document

• Min: 1

• Max: 1

• JSON Generator

• Binary to Document

Incoming documents are first written to a staging file on Snowflake's internal staging area. A temporary table is created on Snowflake with the contents of the staging file. An update operation is then run to update existing records in the target table and/or an insert operation is run to insert new records into the target table.

Output

Document

• Min: 0

• Max: 1

• Mapper

• Snowflake Execute

If an output view is available, then the output document displays the number of input records and the status of the bulk upload.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

• Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

• Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

• Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Label*

Default Value: Snowflake - Bulk Upsert
Example: Load Employee Tables

String

N/A

Specify the name for the instance. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Schema Name

Default Value: N/A
Example: schema_demo

String/Expression/Suggestion

N/A

Specify the database schema name. In case it is not defined, then the suggestion for the Table Name retrieves all tables names of all schemas. The property is suggestible and will retrieve available database schemas during suggest values.

Table Name*

Default Value: N/A
Example: employees_table

String/Expression/Suggestion

N/A

Specify the name of the table to execute bulk load operation on.

Staging location

Default Value: Internal
Example: External

Dropdown list/Expression

N/A

Select the type of staging location that is to be used for data loading:

• External: Location that is not managed by Snowflake. The location should be an AWS S3 Bucket or Microsoft Azure Storage Blob or Google Cloud Storage. These credentials are mandatory while validating the Account.

• Internal: Location that is managed by Snowflake.

Flush chunk size (in bytes)

String/Expression

Appears when you select Input view for Data source and Internal for Staging location.

When using internal staging, data from the input view is written to a temporary chunk file on the local disk. When the size of a chunk file exceeds the specified value, the current chunk file is copied to the Snowflake stage and then deleted. A new chunk file simultaneously starts to store the subsequent chunk of input data. The default size is 100,000,000 bytes (100 MB), which is used if this value is left blank.

File format object

Default Value: None

Example: jsonPath()

String/Expression

N/A

Specify an existing file format object to use for loading data into the table. The specified file format object determines the format type such as CSV, JSON, XML, AVRO, or other format options for data files.

File format type

Default Value: None
Example: CSV

String/Expression/Suggestion

N/A

Specify a predefined file format object to use for loading data into the table. The available file formats include CSV, JSON, XML, and AVRO.

File format option

Default value:  N/A
Example: BINARY_FORMAT=UTF8

String/Expression

N/A

Specify the file format option. Separate multiple options by using blank spaces and commas.

Encryption type

Default Value: None
Example: Server-Side Encryption

Dropdown list

N/A

Specify the type of encryption to be used on the data. The available encryption options are:

• None: Files do not get encrypted.

• Server Side Encryption: The output files on Amazon S3 are encrypted with server-side encryption.

• Server-Side KMS Encryption: The output files on Amazon S3 are encrypted with an Amazon S3-generated KMS key. 

KMS key

Default Value: N/A
Example: <Encrypted>

String/Expression

N/A

Specify the KMS key that you want to use for S3 encryption. Learn more about the KMS key: AWS KMS Overview and Using Server Side Encryption.

Buffer size (MB)

Default Value: 10MB
Example: 20MB

String/Expression

N/A

Specify the data in MB to be loaded into the S3 bucket at a time. This property is required when bulk loading to Snowflake using AWS S3 as the external staging area.

Minimum value: 6 MB

Maximum value: 5000 MB

Specify the column to use for existing entries in the target table.

String

N/A

Delete Upsert Condition when true, causes the case to be executed.

Checkbox

N/A

Select this check box to preserve the case sensitivity of the column names.

• If you do not select Preserve case sensitivity, the input documents are loaded to the target table if the key names in the input documents match the target table column names ignoring the case.

• If you include a second input view, selecting Preserve case sensitivity has no effect on the column names of the target table, because Snap uses the metadata from the second input view.

Manage Queued Queries

Default Value: Continue to execute queued queries when the Pipeline is stopped or if it fails
Example: Cancel queued queries when the Pipeline is stopped or if it fails

Dropdown list

N/A

Select this property to determine whether the Snap should continue or cancel the execution of the queued Snowflake Execute SQL queries when you stop the pipeline.

Load empty strings

Default value:  Selected

Checkbox

N/A

If selected, empty string values in the input documents are loaded as empty strings to the string-type fields. Otherwise, empty string values in the input documents are loaded as null. Null values are loaded as null regardless.

Additional Options

On Error

Default Value: ABORT_STATEMENT
Example: CONTINUE

Dropdown list

N/A

Select an action to perform when errors are encountered in a file. The available actions are:

• ABORT_STATEMENT: Aborts the COPY statement if any error is encountered. The error will be thrown from the Snap or routed to the error view.

• CONTINUE: Continues loading the file. The error will be shown as a part of the output document.

• SKIP_FILE: Skips file if any errors encountered in the file.

• SKIP_FILE_*error_limit*: Skips file when the number of errors in the file exceeds the number specified in Error Limit.

• SKIP_FILE_*error_percent_limit*%: Skips file when the percentage of errors in the file exceeds the percentage specified in Error percentage limit.

Error Limit

Default Value: 0
Example: 3

Integer

Appears when you select SKIP_FILE_*error_limit* for On Error.

Specify the error limit to skip file. When the number of errors in the file exceeds the specified error limit or when SKIP_FILE_number is selected for On Error.

Error Percentage Limit

Default Value: 0
Example: 1

Integer

Appears when you select SKIP_FILE_*error_percent_limit*% 
for On Error.

Specify the percentage of errors  to skip file. If the file exceeds the specified percentage when SKIP_FILE_number% is selected for On Error. 

Snap Execution

Default Value: Execute only
Example: Validate & Execute

Dropdown list

N/A

Select one of the three modes in which the Snap executes. Available options are:

• Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

• Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

• Disabled: Disables the Snap and all Snaps that are downstream from it.

Cannot lookup a property on a null value.

The value referenced in the Key Column field is null.

This Snap does not support values from an upstream input document in the Key columns field when the expression button is enabled.

Update the Snap settings to use an input value from pipeline parameters and run the pipeline again.

Data can only be read from Google Cloud Storage (GCS) with the supplied account credentials (not written to it).

Snowflake Google Storage Database accounts do not support external staging when the Data source is the Input view.

Data can only be read from GCS with the supplied account credentials (not written to it).

Use internal staging if the data source is the input view or change the data source to staged files for Google Storage external staging.