$customHeader
Skip to end of banner
Go to start of banner

AlloyDB Bulk Load

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

In this article

Overview

You can use this Snap to bulk load data into AlloyDB tables. If the target table does not exist in AlloyDB, you can create a table. This Snap supports both binary and document inputs. You can configure the input view in the Views tab.  

Snap Type

The AlloyDB Bulk Load Snap is a Write-type Snap.

Prerequisites

None.

Support for Ultra Pipelines  

Does not support Ultra Pipelines

Limitations

If the target table does not exist and the input is binary, the Snap does not fetch the table's schema from the input. The table's schema must be passed to the Snap through the second input view.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

  • Document

  • Binary

  • Min: 1

  • Max: 2

  • Mapper Snap

  • Copy Snap

  • File Reader

  • AlloyDB Select

The data to be loaded into the AlloyDB table. This Snap has one document input view by default.

A second view can be added for table metadata as a binary or document stream so that the target absent table can be created in the database with a schema similar to the source table. This schema is usually from the output of a database Select Snap.

Output

Document

  • Min: 0

  • Max: 1

  • JSON Formatter

  • File writer

The Snap outputs one document specifying the status, with the records count that are being inserted into the table. Any error occurring during the process is routed to the error view.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current Pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the fieldset.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the fieldset.

  • Upload icon ((blue star) ): Indicates that you can upload files.

Field Name

Field Type

Description

Label*

Default ValueAlloyDB Bulk Load
ExampleLoad Employee Data

String

Specify a unique name for the Snap.

Schema

Default Value: N/A
Example: Employee 

String/Expression

Specify the database schema name in AlloyDB. If you do not specify a schema, then the suggestion for the Table name retrieves all table names of all schemas.

Table name*


Default Value: N/A
ExampleEmployee.PersonnelInformation

String/Expression

Specify the AlloyDB table on which to run the bulk load operation.

Create table if not present


Default ValueDeselected

Checkbox

Select this checkbox to automatically create the target table if it does not exist.

  • If a second input view is configured for the Snap and it contains a document with schema (metadata) of the source table, the Snap creates the new (target) table using the same schema (metadata). However, if the schema comes from a different database, the Snap might fail with Unable to create table: "<table_name>" error due to data type incompatibility.

  • In the absence of a second input view (the schema/metadata document), the Snap creates a table based on the data types of the columns generated from the first row of the input document (first input view).

A newly created table is not visible to subsequent database Snaps during runtime validation. If you want to immediately use the newly updated data you must use a child Pipeline that is invoked through a Pipeline Execute Snap.

Columns

Use this field set to define the columns to be loaded/copied. If you do not configure this field set, then all columns in the input data stream are loaded into the target table. You can also set how you want the Snap to handle null values for each column.

  • Column order is critical when using the binary input view.

  • You can select both Force Non-Null and Force Null simultaneously. In such cases, the Snap converts quoted null strings to null values and unquoted null strings to empty strings.

Column


Default Value: N/A
Exampleemp_name

String/Expression

Specify the name of the column/field. This is a suggestible field and lists all field names in the input.

Force Not-Null


Default ValueDeselected
Example: N/A

Checkbox

Select this checkbox to not match the selected column's value against the null string. The Snap reads empty null strings as zero-length strings rather than nulls, even when they are not quoted.

Force Null


Default ValueDeselected
Example: N/A

Checkbox

Select this checkbox to match the selected column value against the null string, even if it has been quoted. If a match is found, the field value is set to NULL.

By default, when the null string is empty, the Snap converts a quoted empty string into NULL.

Format*


Default ValueText
ExampleCSV

Dropdown list

Choose the data format to be written. This field is applicable only when using the binary input view. 

  • Text - tab-separated-values (default)

  • CSV - comma-separated-values

  • Binary - AlloyDB specific format

Header Provided


Default ValueDeselected
Example: N/A

Checkbox

Select this checkbox to include the input data as a header. Applicable only when using the binary input view and CSV option in the Format field.

Encoding

Default ValueUTF8
Example: Unicode

Dropdown list

Choose one of the following encodings to use. This is limited to the encodings supported by the AlloyDB server:

 Encodings:

ABC

ALT

BIG5

EUC_CN

EUC_JIS_2004

EUC_JP

EUC_KR

EUC_TW

GB18030

GBK

ISO88591

ISO885910

ISO885913

ISO885914

ISO885915

ISO885916

ISO88592

ISO88593

ISO88594

ISO88599

ISO_8859_5

ISO_8859_6

ISO_8859_7

ISO_8859_8

  • ISO_8859_8

  • JOHAB

  • KOI8

  • KOI8R

  • KOI8U

  • LATIN1

  • LATIN2

  • LATIN3

  • LATIN4

  • LATIN5

  • LATIN7

  • LATIN9

  • Mskanji

  • SHIFT_JIS_2004

  • SJIS

  • SQL_ASCII

  • ShiftJIS

  • TCVN

  • TCVN5712

  • UHC

  • UTF8

  • Unicode

  • VSCII

  • WIN

  • WIN1250

  • WIN1251

  • WIN1252

  • WIN1253

  • WIN1254

  • WIN1255

  • WIN1256

  • WIN1257

  • WIN1258

  • WIN866

  • WIN874

  • WIN932

  • WIN936

  • WIN949

  • WIN950

  • Windows932

  • Windows936

  • Windows949

  • Windows950

Snap Execution

Default Value
Example: Validate & Execute

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline runtime without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Snap Pack History

Release

Snap Pack Version

Date

Type

Updates

May 2024

4postgresupgrade26570

-

The JDBC driver for the Google AlloyDB Snap Pack has been upgraded from v42.4.2 to v42.7.2 (Java 8). This upgrade will be part of the GA release on August 14, 2024 (Stable release). As part of this upgrade, the 42.7.2 JDBC driver is bundled with the AlloyDB Snap Pack as the default JDBC driver. Learn more about the behavior changes caused by the driver upgrade.

May 2024

main26341

Stable

Updated the Delete Condition (Truncates a Table if empty) field in the AlloyDB Delete Snap to Delete condition (deletes all records from a table if left blank) to indicate that all entries will be deleted from the table when this field is blank, but no truncate operation is performed.

February 2024

main25112

Stable

Updated and certified against the current SnapLogic Platform release.

November 2023

main23721

Stable

Updated and certified against the current SnapLogic Platform release.

August 2023

main22460

Stable

The AlloyDB Execute Snap now includes a new Query type field. When Auto is selected, the Snap tries to determine the query type automatically.

May 2023

main21015

Stable

Upgraded with the latest SnapLogic Platform release.

February 2023

main19844

Stable

Upgraded with the latest SnapLogic Platform release.

November 2022

main18944

Stable

The AlloyDB Insert Snap now creates the target table only from the table metadata of the second input view when the following conditions are met:

  • The Create table if not present checkbox is selected.

  • The target table does not exist.

  • The table metadata is provided in the second input view.

September 2022

430patches18149

Latest

The AlloyDB Select and AlloyDB Execute Snaps now read NaN values in Numeric columns.

4.30Patches

430patches17841

Latest

Introduced the Google AlloyDB Snap Pack— a fully managed PostgresSQL-compatible database service that you can use for all your database workloads. This Snap Pack offers the following Snaps:


  • No labels