You can use this Snap to create BigQuery Datasets. Datasets are top-level containers that you can use to organize and control access to your tables and views.
Snap Type
BigQuery Dataset Create Snap is a WRITE-type Snap that creates BigQuery Dataset.
Prerequisites
A valid Google BigQuery Account with the required permissions.
Access/Valid permissions to the project to create a dataset
You can set the geographic location only at the time of creating a dataset. After you create the dataset, the location becomes immutable.
Dataset names must be unique for each project.
Known Issues
None.
Snap Views
Type
Format
Number of Views
Examples of Upstream and Downstream Snaps
Description
Input
Document
Min: 0
Max: 1
Mapper
Any BigQuery Snap
The Snap can have at the most one input document that provides the value for the project ID or Dataset ID
Output
Document
Min: 1
Max: 1
File Writer
Mapper
The Snap has a minimum of one output that provides information on whether the dataset is created or not. You can add an input and error views to view the result.
Error
Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.
Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.
Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.
Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
Expression icon (): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon (): Indicates that you can add fields in the fieldset.
Remove icon (): Indicates that you can remove fields from the fieldset.
Specify a unique name for the Dataset that is to be created.
Dataset names must be unique for each project.
Project ID*
Default Value: None Example: case16370
String/Expression
None
Specify the Project ID in which the dataset is to be created.
Dataset ID*
Default Value: None Example: dscreate5
String/Expression
None
Specify the Project ID in which the dataset is to be created.
Detailed Information
Default Value: Deselected Example: Selected
Checkbox
None
Select this checkbox to allow the Snap to display additional fields which can be included while creating the dataset.
Location
Default Value: None Example: Europe - Finland (europe-north1)
String/Expression
Displays when Detailed Information is selected.
Specify the geographic location where you want to create the dataset.
You can set the geographic location only at the time of creating a dataset. After you create the dataset, the location becomes immutable.
Default Table Expiration (in Days)
Default Value: None Example: 10
String/Expression
Displays when Detailed Information is selected.
Specify the number of days after which any new table created in this dataset should be automatically deleted. You can use this field only if you want to create a dataset to store temporary data that is not to be preserved for longer. If you want to preserve the tables for longer, then leave this field blank.
Snap Execution
Default Value: Example: Validate & Execute
Dropdown list
N/A
Select one of the following three modes in which the Snap executes:
Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
Disabled: Disables the Snap and all Snaps that are downstream from it.
Examples
Creating a Dataset and then Deleting the Same
This example Pipeline demonstrates how to create a Dataset by using the Dataset Create Snap at the specified location.
First, configure the Dataset Create Snap to create the dataset as specified below.
On validation, the output displays that the Dataset has been created with the timestamp.
Updated and certified against the current SnapLogic Platform release.
February 2024
main25112
Stable
Updated and certified against the current SnapLogic Platform release.
November 2023
main23721
Stable
Updated and certified against the current SnapLogic Platform release.
August 2023
main22460
Stable
Updated and certified against the current SnapLogic Platform release.
May 2023
433patches22057
Latest
Introduced the Google BigQuery Upsert (Streaming) Snap, which enables you to perform bulk update/insert operations into a BigQuery table from existing tables or any input data stream.
Fixed an issue with the BigQuery Table Create Snap that caused an error to display when fields were separated with a comma and a space.
Fixed an issue with the sorting of the Partitioning time dropdown in the BigQuery Table Create Snap.
February 2023
432patches19840
Latest
Fixed an issue with the Google BigQuery Bulk Load (Streaming) Snap that caused the Table not found message to display even when Create table if not present was selected.
Upgraded with the latest SnapLogic Platform release.
4.28 Patch
428patches15459
Latest
Fixed an issue with Google BigQuery Execute Snap, where the Snap displayed 404 Job not found error when calling a procedure.
Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap where the Snap failed, because the access token expired when it had to wait longer to execute. With this fix, the Snap is reloaded to get refreshed access token.
4.28 Patch
428patches14743
Latest
Fixed an issue with the Google BigQuery Execute Snap, where the Snap displayed an error when the input data contained a table having the record type column and its value was null.
Fixed an issue with the Google BigQuery Write Snap, when the input data contained complex data type columns (such as nested fields) and Create table if not present checkbox was selected.
4.28
main14627
Stable
Upgraded with the latest SnapLogic Platform release.
4.27 Patch
427patches13752
Latest
Upgraded Google BigQuery driver to 1.119.0 version to support time partition intervals by MONTH and YEAR.
4.27 Patch
427patches13615
Latest
Fixed the table truncate 404 error with the Google BigQuery Load (Streaming) Snap by supporting the retry functionality. The Snap now waits in case of an error and retries before loading the data.
4.27 Patch
427patches12691
Latest
Fixed an issue with the Google BigQuery Bulk Load (Cloud Storage) Snap, where the Snap failed with an exception for big query tables. The CreateDisposition is now set conditionally on the basis of the setting in the Create table if not present checkbox.
Batch Size: The number of records batched per request.
Batch Timeout (milliseconds): Time in milliseconds to elapse following which the batch, if not empty, will be processed even though it might be lesser than the given batch size.
4.26
main11181
Stable
Upgraded with the latest SnapLogic Platform release.
4.25
main9554
Stable
Upgraded with the latest SnapLogic Platform release.
4.24
main8556
Stable
Upgraded with the latest SnapLogic Platform release.
Fixed the precision loss inGoogle BigQuery ExecuteSnap outputthat strips millisecond valueswhile retrieving TIMESTAMP values from Google BigQuery tables.
4.22
main6403
Stable
Upgraded with the latest SnapLogic Platform release.
4.21
snapsmrc542
Stable
Upgraded with the latest SnapLogic Platform release.
4.20 Patch
google/bigquery8773
Latest
Fixed the NPE issue with stored procedures and DROP TABLE queries in the Google BigQuery Execute Snap.
4.20
snapsmrc535
Stable
Upgraded with the latest SnapLogic Platform release.
4.19
snaprsmrc528
Stable
Upgraded with the latest SnapLogic Platform release.
4.18
snapsmrc523
Stable
Upgraded with the latest SnapLogic Platform release.
4.17
ALL7402
Latest
Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.
4.17
snapsmrc515
Stable
Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.
4.16
snapsmrc508
Stable
Upgraded with the latest SnapLogic Platform release.
4.15
snapsmrc500
Stable
Upgraded with the latest SnapLogic Platform release.
4.14
snapsmrc490
Stable
Upgraded with the latest SnapLogic Platform release.
4.13
snapsmrc486
Stable
Upgraded with the latest SnapLogic Platform release.
4.12
snapsmrc480
Stable
Added a new propertySchema auto detectin theGoogle BigQuery Bulk Load (Cloud Storage) Snap to support CSV and JSON files where one or more columns in the source file may not contain any values.
4.11
snapsmrc465
Stable
Added new Snap: Google BigQuery Bulk Load (Cloud Storage)
Added new Snap: Google BigQuery Bulk Load (Streaming).
Updated Google Big Query Write Snap with a new Create table if not presentproperty.
4.10 Patch
google/bigquery4046
Latest
Addressed an issue when authenticating with Dynamic OAuth accounts.
4.10
snapsmrc414
Stable
Upgraded with the latest SnapLogic Platform release.
4.9
snapsmrc405
Stable
Upgraded with the latest SnapLogic Platform release.
4.8 Patch
bigquery2952
Latest
Supports refreshing OAuth access tokens during long-running pipeline executions. Fixed an issue with writing small batch sizes and when querying empty dataset tables.
4.8.0 Patch
bigquery2813
Latest
Reload OAuth account from Platform when the access token expires during pipeline execution.
4.8
snapsmrc398
Stable
Upgraded with the latest SnapLogic Platform release.
4.7
snapsmrc382
Stable
Upgraded with the latest SnapLogic Platform release.
4.6
snapsmrc362
Stable
Upgraded with the latest SnapLogic Platform release.
4.5.1
snapsmrc344
Stable
Upgraded with the latest SnapLogic Platform release.
4.4.1
NA
Stable
Upgraded with the latest SnapLogic Platform release.
4.4
NA
Stable
Upgraded with the latest SnapLogic Platform release.
4.3.2
NA
Stable
Resolved the following issues with the Google BigQuery Execute Snap:
throwing binary data in stacktrace and two error messages.
improve error handling for suggestions
improve error handling on bad queries
suggestion bubble missing for Destination table ID
Resolved an issue with Auto refresh token not working in Google BigQuery account.