Parquet Writer
- Kalpana Malladi
In this article
Overview
You can use this Snap to convert documents to the Parquet format and write the data to HDFS, ADL (Azure Data Lake), ABFS (Azure Blob File Storage), WASB (Azure storage), or an S3 bucket. This Snap supports a nested schema such as LIST and MAP. You can also use this Snap to write schema information to the Catalog Insert Snap.
This Snap supports HDFS, ADL (Azure Data Lake), ABFS (Azure Blob File Storage), WASB (Azure storage), and S3 protocols.
Snap Type
The Parquet Writer Snap is a Write-type Snap.
Prerequisites
You must have access and permission to write to HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Storage Gen 2), WASB (Azure storage), or AWS S3.
Support for Ultra Pipelines
Works in Ultra Pipelines.
Supported Versions
The Parquet Writer Snap is tested against Windows Server 2008, 2010, and 2012.
Limitations
Auto schema generation in this Snap excludes null fields. For example, if the Snap receives ten input documents during preview execution, and four of these documents contain null values for certain fields in all instances, those four fields are disregarded during schema generation. The schema only includes fields with at least one non-null value among the preview input documents.
"Generate template" is unsupported for a nested structure like MAP and LIST type. Generate template is a link within the schema editor accessed through the Edit Schema button.
All expression Snap properties can be evaluated (when the '=' button is pressed) from pipeline parameters only, not from input documents from upstream Snaps. Input documents are data to be formatted and written to the target files.
The security model configured for the Groundplex (SIMPLE or KERBEROS authentication) must match the security model of the remote server. Due to the limitations of the Hadoop library, we can only create the necessary internal credentials to configure the Groundplex.
Parquet Snaps work well in a Linux environment. However, due to limitations in the Hadoop library on Windows, their functioning in a Windows environment may not always be as expected. We recommend you use a Linux environment for working with Parquet Snaps.
To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:
Download
hadoop.dllandwinutils.exewinutils/hadoop-3.2.2/bin at master · cdarlint/winutils (SnapLogic’s Hadoop version is 3.2.2)Create a temporary directory.
Place the
hadoop.dllandwinutils.exefiles in this path:C:\\hadoop\binSet the environment variable
HADOOP_HOMEto point toC:\\hadoopAdd
C:\hadoop\binto the environment variable PATH as shown below:Add the JVM options in the Windows Snaplex:
jcc.jvm_options= -Djava.library.path=C:\\hadoop\binIf you already have an existing
jvm_options, then add:"-Djava.library.path=C:\\hadoop\bin"after the space.
For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\hadoop\binRestart the JCC for configurations to take effect.
Known Issues
Learn more about the Azure Storage library upgrade.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
|---|---|---|---|---|
Input | Document
|
| Mapper |
|
Output | Document |
| Mapper | A document with a filename for each Parquet file written. For example: |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:
Learn more about Error handling in Pipelines. | |||
Snap Settings
Asterisk ( * ): Indicates a mandatory field.
Suggestion icon (Emoji :Suggest_Icon: ): Indicates a list that is dynamically populated based on the configuration.
Expression icon (Emoji :Expression_Enabler: ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon ( ): Indicates that you can add fields in the field set.
Remove icon ( Emoji :Remove_icon: ): Indicates that you can remove fields from the field set.
Upload icon (Emoji :Upload_icon: ): Indicates that you can upload files.
Field Name | Field Type | Description |
|---|---|---|
Label* Default Value: Parquet Writer | String | Specify a unique name for the Snap. |
Directory
Default Value: hdfs://<hostname>:<port>/
| String/Expression/Suggestion | Specify the file path to a directory to read data from. The path must be in the following format: All files within the directory must be Parquet formatted. The following file storage systems are supported:
SnapLogic automatically appends The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property. |
File Filter
Default Value: *
| String/Expression | Specify the Glob file pattern. The following rules are used to interpret glob patterns: - The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period. - The ** characters match zero or more characters across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory. - The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension. - The \ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash, and \{ matches a left brace, for example. - The ! character is used to exclude matching files from the output. - The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'. - Within a bracket expression, the '*', '?', and '\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating. - The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'. - Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name ".login". - All other characters match themselves. Examples: '*.csv' matches all files with a csv extension in the current directory only. '**.csv' matches all files with a csv extension in the current directory and in all its subdirectories. *[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP. |
File Default Value: N/A
| String/Expression/Suggestion | Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". |
Hive Metastore URL Default Value: N/A | String/Expression | Specify the URL of the Hive Metastore to assist in setting the schema along with the database and table setting. If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where you define the schema. |
Database Default value: N/A | String/Expression/Suggestion | Specify the Hive Metastore database which holds the schema for the outgoing data.
|
Table
Default value: N/A | String/Expression/Suggestion | Specify the table whose schema should be used for formatting the outgoing data.
|
Fetch Hive Schema at Runtime
Default value: Deselected | Checkbox | Select this checkbox to fetch the schema from the Metastore table before writing. The Snap fails to write if it cannot make connection to the metastore or the table does not exist during the Pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked. |
Edit Schema
| Button | Specify a valid Parquet schema that describes the data. After defining the message type, a list of fields are given. A field is comprised of a repetition, a type, and the field name. Available repetitions are required, optional, and repeated.
These types can be annotated with a logical type to specify how the application should interpret the data. The Logical types include:
Unsigned types - may be used to produce smaller in-memory representations of the data. If the stored value is larger than the maximum allowed by int32 or int64, then the behavior is undefined.
DECIMAL(precision, scale) - used to describe arbitrary-precision signed decimal numbers of the form value * 10^(-scale) to the given precision. The annotation can be with int32, int64, fixed_len_byte_array, binary. See the Parquet documentation for limits on precision that can be given. DATE - used with int32 to specify the number of days since the Unix epoch, 1 January 1970 Note: This Snap supports only the following date format: yyyy-MM-dd.
a number in months a number in days a number in milliseconds
The following is an example of a schema using all the primitive and some examples of logical types: message document {
# Primitive Types
optional int64 32_num;
optional int64 64_num;
optional boolean truth;
optional binary message;
optional float pi;
optional double e;
optional int96 96_num;
optional fixed_len_byte_array (1) one_byte;
# Logical Types
optional binary snowman (UTF8);
optional int32 8_num (INT_8);
optional int32 16_num (INT_16);
optional int32 u8_num (UINT_8);
optional int32 u16_num (UINT_16);
optional int32 u32_num (UINT_32);
optional int64 u64_num (UINT_64);
optional int32 dec_num (DECIMAL(5,2));
optional int32 jan7 (DATE);
optional int32 noon (TIME_MILLIS);
optional int64 jan7_epoch (TIMESTAMP_MILLIS);
optional binary embedded (JSON);
}
|
Compression*
| Dropdown list | Select the type of compression to use when writing the file. The available options are:
|
Partition by
Default Value: N/A | String/Suggestion | Specify or select the key which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Refer to the 'Partition by' example below for an illustration. |
Azure SAS URI Properties | Shared Access Signatures (SAS) properties of the Azure Storage account. | |
SAS URI | String/Expression | Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account. |
Decimal Rounding Mode
Default Value: Half up Example: Up | Dropdown list | Select the required rounding method for decimal values when they exceed the required number of decimal places. The available options are:
|
Snap Execution | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Troubleshooting
Error | Reason | Resolution |
|---|---|---|
Unable to connect to the Hive Metastore. | This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore. | Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:
|
Parquet Snaps may not work as expected in the Windows environment. | Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected. | To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:
|
| Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected. | To resolve this issue, follow these steps:
|
Additional Information
Write to S3 files with HDFS version CDH 5.8 or later
When running the HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:
Go to HDFS configuration.
In Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, add an entry with the following details:
Name: fs.s3a.threads.max
Value: 15
Click Save.
Restart all the nodes.
Under Restart Stale Services, select Re-deploy client configuration.
Click Restart Now.
Temporary Files
During execution, data processing on Snaplex nodes occurs principally in-memory as streaming and is unencrypted. When larger datasets are processed that exceed the available compute memory, the Snap writes Pipeline data to local storage as unencrypted to optimize the performance. These temporary files are deleted when the Snap/Pipeline execution completes. You can configure the temporary data's location in the Global properties table of the Snaplex's node properties, which can also help avoid Pipeline errors due to the unavailability of space. For more information, see Temporary Folder in Configuration Options.
Examples
Parquet Writer Snap with the Second Input View
The following example pipeline demonstrates the usage of the second input view to receive the table metadata. You can provide the Hive Metastore information in the Hive Metastore URL field, wherein a single input view is sufficient.
The Parquet Writer Snap configuration with Directory path:
| The Parquet Writer Snap View:
|
The Hive Execute Snap with the table metadata information to pass to the second input view of the Parquet Writer Snap: | The Hive Execute Snap Output: |
Inserting and Querying Custom Metadata from the Flight Metadata Table
The Pipeline in this zipped example, MetadataCatalog_Insert_Read_Example.zip, demonstrates how you can:
Use the Catalog Insert Snap to update metadata tables.
Use the Catalog Query Snap to read the updated metadata information.
In this example:
We import a file containing the metadata.
We create a parquet file using the data in the imported file
We insert metadata that meets specific requirements into a partition in the target table.
We read the newly-inserted metadata using the Catalog Query Snap.
Downloads
Snap Pack History
Related Content
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.