Overview
The Catalog Delete Snap enables you to delete the partitions within a specific Table Asset in the SnapLogic Data Catalog.
Input and Output
- Expected input: A document containing the metadata to be deleted from the Data Catalog.
- Expected output: A document stream containing status messages on the result of the delete operation.
- Expected upstream Snaps: Required. A Snap that offers document streaming data. Examples: Parquet Writer and Mapper.
- Expected downstream Snaps: Any Snap that accepts binary streaming data in its input view. Examples: Mapper and File Writer.
Prerequisites
Write access to the SnapLogic Data Catalog.
Configuring Accounts
Accounts are not used with this Snap.
Configuring Views
Input | This Snap has exactly one document input view. |
|---|---|
| Output | This Snap has exactly one document output view. |
| Error | This Snap has one document error view. |
Troubleshooting
None.
Limitations and Known Issues
None.
Modes
- Ultra pipelines: Does not work in Ultra pipelines.
- Spark mode: Supports Spark mode.
Snap Settings
| Label | Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline. |
|---|---|
| Table Name | Enter the name of the target Table Asset in the Data Catalog. You can either enter this information manually or you can select the table from the suggestible drop-down. |
| Data Location | Enter the location of the source data from which the Table Asset in the Data Catalog originates. |
| Partition Keys | Add the partition keys which you want to delete. You list the partition keys by specifying Key Column and Key Value combinations to identify the precise row and column from which to create a partition. Default value: None |
| Key Name | Specify the key you want to use to locate a specific partition in a metadata table. Example: Default value: None |
| Operator | Specify the operator to use to locate the table partition. Example: != Default value: = |
| Key Value | Specify the value to use to locate the table partition. Example: 10 Default value: None |
Snap execution | Select one of the three modes in which the Snap executes. Available options are:
|
Example
Pipeline
The following Pipeline demonstrates how to overwrite an existing partition by the functions of the three Data Catalog Service Snaps: Catalog Query, Catalog Delete, and Catalog Insert. Because you cannot edit the Data Catalog in Manager, you can use this Pipeline as a pattern for overwriting partitions in the Data Catalog.
The Catalog Query Snap queries for a partition, age, in the Asset named table1.
The Catalog Delete Snap deletes the partition, PK_1, indicated by the Key Hame and Key Value.
To replace this metadata, a JSON Generator Snap is used to generate a document with the metadata name, age, and height.
The data is passed through the Parquet Writer, which writes the JSON document to an S3 bucket.
The Catalog Insert Snap populates the table with the new metadata.
You can view the metadata inserted in the Data Catalog from the resulting operation when you open the table1 in Manager > Projects > Assets:
In Manager, click table1, and click Show Table Schema.
Download