$customHeader
Skip to end of banner
Go to start of banner

Google BigQuery Google Service Account

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 34 Current »

In this article

Overview

Use this SnapLogic Account type to connect Snaps to Google BigQuery using a Google service account.

Prerequisites

  • Existing Google Service Account with P12 file containing the private key. Learn how to import P12 File into the Google Service Account.

  • JDK with Keytool utility to set up the Keystore. 

Limitations and Known Issues

None.

Account Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ( (blue star) ): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ( (blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the field set.

  • Remove icon ( (blue star) ): Indicates that you can remove fields from the field set.

Field Name

Field Type

Description

Label*

Default Value: None
ExampleService Account

String/Expression

Specify a unique label for the account.

Properties for Google BigQuery

Account ID*   

Default Value: None
Example: 108275755

Integer/Expression

Specify the ID of the service account available in the Google web console.

Private Key ID

Default Value: N/A
Example: 4fb718fd54de73f0

String/Expression

Specify the ID of the private key available in the Google web console.

Email ID*

Default Value: N/A
Example: test1@0.iam.gserviceaccount.com

String/Expression

Specify the email id of the service account available in the Google web console.

Keystore URI*

Default Value: N/A
Example: g_bigq

String/Expression

Specify the location of the Keystore that contains the private key you use. You can upload the file using the Upload icon.

Keystore Passphrase*

String/Expression

Specify the passphrase to access the Keystore.

Private Key Alias*

Default Value: N/A
Example: privatekey

String/Expression

Specify the alias to identify the private key. This is a suggestible field and will lists all the available aliases in the key store.

Properties for Cloud Storage

Account ID 

Default Value: N/A
Example: 108275755

Integer/Expression

Specify the ID of the service account available in the Google web console.

Private Key ID 

Default Value: N/A
Example: 4fb718fd54de73f0

String/Expression

Specify the ID of the private key available in the Google web console.

Email ID 

Default Value: None
Example: test1@0.iam.gserviceaccount.com

String/Expression

Specify the email id of the the service account available in the Google web console.

Keystore URI

Default Value: N/A
Example: test/g_bigquery

String/Expression

Specify the location of the Keystore that contains the private key to be used. Click the browse icon to load into/select from SLDB.

Keystore Passphrase

Default Value: N/A
Example: Encrypted value

String/Expression

Specify the passphrase to access the Keystore.

Private Key Alias 

Default Value: N/A
Example: privatekey

String/Expression

Specify the alias to identify the private key. This is a suggestible field and will list all the aliases in the key store.

Default Standard SQL

Default Value: Deselected

Checkbox

Select this checkbox if you want to default the dialect in the Query field in the Snaps using this account to Standard SQL.

To override this default setting use the #legacySQL tag within the Query field of the Snap to treat it as a Legacy SQL query.


Importing P12 File

Key Steps

  1. Create a new Keystore.

  2. Empty the Keystore.

  3. Verify if the Keystore is empty.

  4. Import the P12 file.

  5. Check the Keystore is non-empty.

  6. Change the alias, if needed.

  7. Verify a new alias.

To perform the above steps, open a Command Prompt window. The steps along with the requisite syntax are described in detail below:

Step 1 - Create New Keystore

  1.  Execute the following command at the Command Prompt.
    keytool -genkey -alias <some alias> -keyalg RSA -keysize 2048 -keystore <keystore name> Example: keytool -genkey -alias demoalias1 -keyalg RSA -keysize 2048 -keystore demokeystore1

  2. Specify the follow-up questions for First and Last name, Organizational unit name, Organization name, City/Locality, State/Province, and Two-letter country code.

  3. Click Yes in the Details confirmation dialog box.

  4. Enter the key password for the provided alias.

  5. Press Enter.

The same key password will be used in the subsequent steps.

Step 2 - Empty the Keystore

  1. Execute the following command at the Command Prompt:
    keytool -delete -alias <alias name> -keystore <keystore name> Example: keytool -delete -alias demoalias1 -keystore demokeystore1

  2. Enter the password as given in Step 1.

Step 3 - Verify if Keystore is Empty

  1. Execute the following command at the Command Prompt:
    keytool -v -list -keystore <keystre name> Example: keytool -v -list -keystore demokeystore1

  2. Enter password as given in Step 1.
    The response should be "Your keystore contains 0 entries".

Step 4 - Import the P12 file

  1. Execute the following command at the Command Prompt:
    keytool -v -importkeystore -srckeystore <p12file.p12> -srcstoretype PKCS12 -destkeystore <keystore name> -deststoretype JKS -destkeypass <password for private key (should be same as keystore passphrase)> Example: keytool -v -importkeystore -srckeystore sample_p12_file.p12 -srcstoretype PKCS12 -destkeystore demokeystore1 -deststoretype JKS -destkeypass demoPassword1

  2. Enter the destination Keystore password given in Step 1.

  3. Enter the source Keystore password that was obtained while generating P12 file.

The response should be "Import command completed: 1 entries successfully imported, 0 entries failed or canceled".

Step 5 - Check Keystore to be Non-empty

  1. Execute the following command at the Command Prompt:

    keytool -v -list -keystore <keystre name>
    Example: 
    keytool -v -list -keystore demokeystore1
  2. Enter password as given in Step 1. The response should show:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

Alias name: privatekey

Creation date: 12 Oct, 2017

Entry type: PrivateKeyEntry

The Keystore is ready with one entry having a private key imported from the P12 file. The alias for that private key is privatekey.

Steps 6 & 7 are needed if you want to change the alias for the imported private key.

Step 6 - Change Alias if needed

  • Execute the following command at the Command Prompt:

keytool -changealias -keystore <keystore name> -alias <current alias> -destalias <new alias>
Example: 
keytool -changealias -keystore demokeystore1 -alias privatekey -destalias first_entry

The response will be blank if the change is successful.

Step 7 - Verify New Alias

Repeat Step 5 and ensure the alias name is what is given in Step 6. The Keystore "demokeystore1" is ready to be used in the snaps with the passphrase given in Step 1.


Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
May 2024main26341 StableUpdated and certified against the current SnapLogic Platform release.
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.

August 2023

main22460

 


Stable

Updated and certified against the current SnapLogic Platform release.

May 2023433patches22057  Latest

Introduced the Google BigQuery Upsert (Streaming) Snap, which enables you to perform bulk update/insert operations into a BigQuery table from existing tables or any input data stream.

May 2023433patches21955 LatestFixed an issue with the GBQ-Google Service Account that caused an input stream to remain open.
May 2023main21015 StableUpdated and certified to be compatible with the August SnapLogic Platform release.
February 2023432patches20962 LatestFixed an intermittent null pointer exception that occurred in the BigQuery Write Snap.
February 2023

432patches20298

 Latest
February 2023432patches19840 Latest

Fixed an issue with the Google BigQuery Bulk Load (Streaming) Snap that caused the Table not found message to display even when Create table if not present was selected.

February 2023main19844 Stable
November 2022431patches19301 LatestThe Google BigQuery Bulk Load (Streaming) Snap works as expected, with no active Timer threads remaining when the Pipeline execution fails. 
November 2022main18944 StableUpgraded with the latest SnapLogic Platform release.
August 2022main17386 Stable
4.29main15993 Stable

Upgraded with the latest SnapLogic Platform release.

4.28 Patch428patches15459 Latest
  • Fixed an issue with Google BigQuery Execute Snap, where the Snap displayed 404 Job not found error when calling a procedure.
  • Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap where the Snap failed, because the access token expired when it had to wait longer to execute. With this fix, the Snap is reloaded to get refreshed access token.

4.28 Patch428patches14743 Latest
  • Fixed an issue with the Google BigQuery Execute Snap, where the Snap displayed an error when the input data contained a table having the record type column and its value was null.
  • Fixed an issue with the Google BigQuery Write Snap, when the input data contained complex data type columns (such as nested fields) and Create table if not present checkbox was selected.

4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches13752 Latest

Upgraded Google BigQuery driver to 1.119.0 version to support time partition intervals by MONTH and YEAR.

4.27 Patch427patches13615 LatestFixed the table truncate 404 error with the Google BigQuery Load (Streaming) Snap by supporting the retry functionality. The Snap now waits in case of an error and retries before loading the data.
4.27 Patch427patches12691 Latest

Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap, where the Snap failed with an exception for big query tables. The CreateDisposition is now set conditionally on the basis of the setting in the Create table if not present checkbox.

4.27

main12833

Stable

Enhanced the Google BigQuery Bulk Load (Cloud Storage) Snap with the following batching and retry properties to process input records:

  • Batching: Processes the input records in batches.

  • Batch Size: The number of records batched per request.

  • Batch Timeout (milliseconds): Time in milliseconds to elapse following which the batch, if not empty, will be processed even though it might be lesser than the given batch size.

4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
StableUpgraded with the latest SnapLogic Platform release.
4.23main7430
 
Stable
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542

 

StableUpgraded with the latest SnapLogic Platform release.
4.20 Patch google/bigquery8773 Latest

Fixed the NPE issue with stored procedures and DROP TABLE queries in the Google BigQuery Execute Snap.

4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19snaprsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18snapsmrc523
 
StableUpgraded with the latest SnapLogic Platform release.
4.17ALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Stable

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508
 
StableUpgraded with the latest SnapLogic Platform release.
4.15snapsmrc500
 
StableUpgraded with the latest SnapLogic Platform release.
4.14snapsmrc490
 
StableUpgraded with the latest SnapLogic Platform release.
4.13

snapsmrc486

 
StableUpgraded with the latest SnapLogic Platform release.
4.12

snapsmrc480

 
Stable

Added a new property Schema auto detect in the Google BigQuery Bulk Load (Cloud Storage) Snap to support CSV and JSON files where one or more columns in the source file may not contain any values. 

4.11snapsmrc465
 
Stable
  • Added new Snap: Google BigQuery Bulk Load (Cloud Storage)
  • Added new Snap: Google BigQuery Bulk Load (Streaming).
  • Updated Google Big Query Write Snap with a new Create table if not present property.
4.10 Patch google/bigquery4046 Latest

Addressed an issue when authenticating with Dynamic OAuth accounts.

4.10

snapsmrc414

 
StableUpgraded with the latest SnapLogic Platform release.
4.9snapsmrc405
 
StableUpgraded with the latest SnapLogic Platform release.
4.8 Patchbigquery2952 Latest

Supports refreshing OAuth access tokens during long-running pipeline executions. Fixed an issue with writing small batch sizes and when querying empty dataset tables.

4.8.0 Patchbigquery2813 Latest

Reload OAuth account from Platform when the access token expires during pipeline execution.

4.8

snapsmrc398

 
StableUpgraded with the latest SnapLogic Platform release.
4.7

snapsmrc382

 
StableUpgraded with the latest SnapLogic Platform release.
4.6snapsmrc362
 
StableUpgraded with the latest SnapLogic Platform release.
4.5.1

snapsmrc344

 
StableUpgraded with the latest SnapLogic Platform release.
4.4.1NA StableUpgraded with the latest SnapLogic Platform release.
4.4NA StableUpgraded with the latest SnapLogic Platform release.
4.3.2NA Stable
  • Resolved the following issues with the Google BigQuery Execute Snap:
    • throwing binary data in stacktrace and two error messages.
    • improve error handling for suggestions
    • improve error handling on bad queries
    • suggestion bubble missing for Destination table ID
  • Resolved an issue with Auto refresh token not working in Google BigQuery account.


Related Content

  • No labels