Parquet Parser
In this article
Overview
You can use this Snap to read the binary (Parquet) data from the input and write the document data to the output.
Snap Type
The Parquet Parser Snap is a Parse-type Snap.
Prerequisites
None.
Support for Ultra Pipelines
Works in Ultra Pipelines.
Limitations and Known Issues
None.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Binary
|
|
| Requires binary (Parquet) input to convert it into the document. |
Output | Document
|
|
| Writes the document data as output. The second output view provides the schema model of the parquet file. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in pipelines. |
Snap Settings
Asterisk ( * ): Indicates a mandatory field.
Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon ( ): Indicates that you can add fields in the field set.
Remove icon ( ): Indicates that you can remove fields from the field set.
Upload icon ( ): Indicates that you can upload files.
Field Name | Field Type | Description |
---|---|---|
Label*
Default Value: Parquet Parser | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.
|
Ignore empty data
Default Value: Selected | Checkbox | When you select this checkbox, the Snap does not generate any output. If you deselect this checkbox, the Snap generates an empty output document when you pass either an empty input stream or an input stream with CSV headers. |
Snap execution Default Value: Validate & Execute | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Example Schema Model for the Parquet Parser Second Output View
[
{
"schemaMap": {
"AUTOSYNC_PRIMARYKEY": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"AUTOSYNC_VALUEHASH": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"AUTOSYNC_CURRENTRECORDFLAG": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"AUTOSYNC_EFFECTIVEBEGINTIME": {
"logicalType": "TIMESTAMP(MILLIS,true)",
"repetition": "OPTIONAL",
"primitiveType": "INT64",
"typeLength": 0
},
"AUTOSYNC_EFFECTIVEENDTIME": {
"logicalType": "TIMESTAMP(MILLIS,true)",
"repetition": "OPTIONAL",
"primitiveType": "INT64",
"typeLength": 0
},
"ID1": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID2": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID3": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID4": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID5": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID6": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID7": {
"logicalType": "STRING",
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID8": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "BINARY",
"typeLength": 0
},
"ID9": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID10": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID11": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID12": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID13": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID14": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
},
"ID15": {
"logicalType": "DATE",
"repetition": "OPTIONAL",
"primitiveType": "INT32",
"typeLength": 0
},
"ID16": {
"logicalType": "TIMESTAMP(MILLIS,true)",
"repetition": "OPTIONAL",
"primitiveType": "INT64",
"typeLength": 0
},
"ID17": {
"logicalType": "TIMESTAMP(MILLIS,true)",
"repetition": "OPTIONAL",
"primitiveType": "INT64",
"typeLength": 0
},
"ID18": {
"logicalType": "TIMESTAMP(MILLIS,true)",
"repetition": "OPTIONAL",
"primitiveType": "INT64",
"typeLength": 0
},
"ID100": {
"logicalType": null,
"repetition": "OPTIONAL",
"primitiveType": "DOUBLE",
"typeLength": 0
}
},
"schema": "message document {\n optional binary AUTOSYNC_PRIMARYKEY (STRING);\n optional binary AUTOSYNC_VALUEHASH (STRING);\n optional binary AUTOSYNC_CURRENTRECORDFLAG (STRING);\n optional int64 AUTOSYNC_EFFECTIVEBEGINTIME (TIMESTAMP(MILLIS,true));\n optional int64 AUTOSYNC_EFFECTIVEENDTIME (TIMESTAMP(MILLIS,true));\n optional double ID1;\n optional binary ID2 (STRING);\n optional binary ID3 (STRING);\n optional binary ID4 (STRING);\n optional binary ID5 (STRING);\n optional binary ID6 (STRING);\n optional binary ID7 (STRING);\n optional binary ID8;\n optional double ID9;\n optional double ID10;\n optional double ID11;\n optional double ID12;\n optional double ID13;\n optional double ID14;\n optional int32 ID15 (DATE);\n optional int64 ID16 (TIMESTAMP(MILLIS,true));\n optional int64 ID17 (TIMESTAMP(MILLIS,true));\n optional int64 ID18 (TIMESTAMP(MILLIS,true));\n optional double ID100;\n}\n"
}
]
Example
To understand the function of Parquet Parser Snap, refer to the example, Transform Document Data into Parquet Format and Vice Versa.
Snap Pack History
Related Content
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.