Parquet Formatter
In this article
Overview
You can use this Snap to read both the input document data and write the data in the binary (Parquet) format to the output.
Snap Type
The Parquet Formatter Snap is a Format-type Snap.
Prerequisites
None.
Support for Ultra Pipelines
Does not work in Ultra Pipelines.
Limitations and Known Issues
None.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document
|
|
| Requires document data as input. You can override the schema setting by inserting an object like this into the second input view. |
Output | Binary
|
|
| Writes the document data in the binary (Pprquet) format to the output. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
Snap Settings
Asterisk ( * ): Indicates a mandatory field.
Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon ( ): Indicates that you can add fields in the field set.
Remove icon ( ): Indicates that you can remove fields from the field set.
Upload icon ( ): Indicates that you can upload files.
Field Name | Field Type | Description |
---|---|---|
Label*
Default Value: Parquet Formatter | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.
|
Edit Schema
| Button | Specify a valid Parquet schema that describes the data. The following is an example of a schema using all the primitive and some examples of logical types: message document {
# Primitive Types
optional int64 32_num;
optional int64 64_num;
optional boolean truth;
optional binary message;
optional float pi;
optional double e;
optional int96 96_num;
optional fixed_len_byte_array (1) one_byte;
# Logical Types
optional binary snowman (UTF8);
optional int32 8_num (INT_8);
optional int32 16_num (INT_16);
optional int32 u8_num (UINT_8);
optional int32 u16_num (UINT_16);
optional int32 u32_num (UINT_32);
optional int64 u64_num (UINT_64);
optional int32 dec_num (DECIMAL(5,2));
optional int32 jan7 (DATE);
optional int32 noon (TIME_MILLIS);
optional int64 jan7_epoch (TIMESTAMP_MILLIS);
optional binary embedded (JSON);
}
"Generate template" does not support nested structures like MAP and LIST type. |
Compression
Default Value: NONE Example: SNAPPY | Dropdown list | Choose the type of compression to use when writing the file. The available options are:
|
Decimal rounding mode
Default Value: Half up Example: Up | Dropdown list | Choose the required rounding method for decimal values when they exceed the required number of decimal places. The following are the available options:
|
Snap execution Default Value: Validate & Execute | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Schema
{
"schema": "message document
{\n optional binary AUTOSYNC_PRIMARYKEY (STRING);
\n optional binary AUTOSYNC_VALUEHASH (STRING);
\n optional binary AUTOSYNC_CURRENTRECORDFLAG (STRING);
\n optional int64 AUTOSYNC_EFFECTIVEBEGINTIME (TIMESTAMP(MILLIS,true));
\n optional int64 AUTOSYNC_EFFECTIVEENDTIME (TIMESTAMP(MILLIS,true));
\n optional double ID1;\n optional binary ID2 (STRING);
\n optional binary ID3 (STRING);
\n optional binary ID4 (STRING);\n optional binary ID5 (STRING);
\n optional binary ID6 (STRING);\n optional binary ID7 (STRING);
\n optional binary ID8;\n optional double ID9;
\n optional double ID10;
\n optional double ID11;\n optional double ID12;
\n optional double ID13;\n optional double ID14;
\n optional int32 ID15 (DATE);\n optional int64 ID16 (TIMESTAMP(MILLIS,true));
\n optional int64 ID17 (TIMESTAMP(MILLIS,true));
\n optional int64 ID18 (TIMESTAMP(MILLIS,true));
\n optional double ID100;
\n}
\n"
}
Examples
Transform document data into Parquet format and vice-versa
This example demonstrates how to convert the input document data to parquet and parquet data back to document output.
Step 1: Configure the JSON Generator Snap with input data.
Step 2: Configure the Parquet Formatter Snap with the schema for the input document data.
Step 3: Configure the Parquet Parser Snap. On validation, the Snap reverts the Parquet data to document data.
Parquet Parser Configuration | Parquet Parser Output |
---|---|
|
|