Catalog Query

On this Page

Overview

The Catalog Query Snap enables you to read the metadata associated with a specific table in the SnapLogic data catalog.

Expected Input and Output

  • Expected Input: A document stream containing information related to the metadata table that must be queried.
  • Expected Output: A document stream containing the contents of the the metadata table that must be queried.
  • Expected Upstream Snaps: Any Snap that offers document streaming data. Examples: Catalog Insert, Mapper.
  • Expected Downstream Snaps: Required. Any Snap that accepts document streaming data in its input view. Examples: JSON Formatter, Catalog Insert.

Prerequisites

Read access to the SnapLogic metadata catalog.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has at most one document input view.
OutputRequired. This Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None.

Known Issues

  • Does not work in Ultra pipelines.



Snap Settings


LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.
Table Name

The location and name of the table that you want to query.

Structure: /<Org>/<Project>/<Table_Name>

Example: /TestOrg/shared/Docs_Test_Table

Default value: None.

Partition Keys

The partition keys in the target table from which you want to extract metadata 

Key Name

The key you want to use to query a specific partition in a metadata table.

Example: airline_code

Default value: None

Operator

The operator that you want to use in the condition required to locate the table partition.

Example: !=

Default value: =

Key Value

The value that you want to use in the condition required to locate the table partition.

Example: 10

Default value: None

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Examples


Inserting and Querying Custom Metadata from the Flight Metadata Table

The Pipeline in this zipped example, MetadataCatalog_Insert_Read_Example.zip, demonstrates how you can:

  • Use the Catalog Insert Snap to update metadata tables.
  • Use the Catalog Query Snap to read the updated metadata information.

In this example:

  1. We import a file containing the metadata.
  2. We create a parquet file using the data in the imported file
  3. We insert metadata that meets specific requirements into a partition in the target table.
  4. We read the newly-inserted metadata using the Catalog Query Snap.


 Understanding the Pipeline

The Pipeline is designed as follows:

The File Reader Snap read flight statistics and the JSON Parser Snap parses the data into a JSON file.

The Parquet Writer Snap creates a Parquet file with the data of the JSON file, in an S3 database.

The output of the Parquet Writer Snap includes the schema of the file. This is the metadata that must be included into the catalog.

The Catalog Insert Snap picks up the schema from the Parquet file and associates it with a specific partition in the target table. It also adds a custom property to the partition.

Once the Snap completes execution, the table is inserted into the metadata catalog and you can view the table in the SnapLogic Manager.

To view the table, navigate to the Project where you have created the Pipeline, click the Table tab, and then click the new table created after executing the Pipeline. This displays the table. Click Show schema to view the metadata.

The Schema view does not display the custom metadata that you inserted into the partition. Use the Catalog Query Snap to view all the updates made by the Catalog Insert Snap.

Download this ZIP file.

 How to use the Sample ZIP File

Working with the Sample ZIP File

This ZIP file contains two files:

  • Metadata_Catalog_Insert_Read.slp
  • AllDataTypes.json

To import this Pipeline:

  1. Download the ZIP file and extract its contents into a local directory.
  2. Import the Metadata_Catalog_Insert_Read.SLP Pipeline into a SnapLogic project.
  3. Open the Pipeline and click the File Reader Snap.
  4. In the File Reader Settings popup, use the  button to import and read the AllDataTypes.json file.
  5. Your Pipeline and test data are now ready. Review the other steps listed out in this example before validating or executing this Pipeline.


Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide Pipeline parameters as applicable.

  File Modified

File MetadataCatalog_Insert_Read_Example.slp

Feb 16, 2022 by Subhajit Sengupta

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateTypeUpdates
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.

August 2023

main22460

 

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 StableUpgraded with the latest SnapLogic Platform release.
August 2022main17386
 
StableUpgraded with the latest SnapLogic Platform release.
4.29

main15993

 
StableUpgraded with the latest SnapLogic Platform release.
4.28main14627
 
StableUpgraded with the latest SnapLogic Platform release.

4.27

main12833

 

Stable

Upgraded with the latest SnapLogic Platform release.
4.26main11181
 
StableUpgraded with the latest SnapLogic Platform release.
4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
StableUpgraded with the latest SnapLogic Platform release.
4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542-StableUpgraded with the latest SnapLogic Platform release.
4.20snaprsmrc528-StableUpgraded with the latest SnapLogic Platform release.
4.19snaprsmrc528-StableUpgraded with the latest SnapLogic Platform release.
4.18snapsmrc523-StableUpgraded with the latest SnapLogic Platform release.
4.17 PatchALL7402-Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.16

snapsmrc508-Stable

New Snap. Added the Catalog Delete Snap to remove tables and table partitions from the Data Catalog. 

4.15snapsmrc500-Stable

New Snap Pack: Introducing the Data Catalog Snap Pack with the following Snaps: Catalog Insert and Catalog Query.