In this article
Overview
You can use this Snap to convert documents to the Parquet format and write the data to HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Storage Gen 2), WASB (Azure storage), or an S3 bucket. This Snap supports a nested schema such as LIST and MAP. You can also use this Snap to write schema information to the Catalog Insert Snap.
This Snap supports HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Storage Gen 2), WASB (Azure storage), and S3 protocols.
Snap Type
The Parquet Writer Snap is a Write-type Snap.
Prerequisites
You must have access and permission to write to HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Storage Gen 2), WASB (Azure storage), or AWS S3.
Support for Ultra Pipelines
Works in Ultra Pipelines.
Supported Versions
The Parquet Writer Snap is tested against Windows Server 2008, 2010, and 2012.
Limitations
Auto schema generation in this Snap involves the exclusion of null fields. For example, if the Snap receives ten input documents during preview execution, and four of these documents contain null values for certain fields in all instances, those four fields are disregarded during schema generation. The schema only includes fields with at least one non-null value among the preview input documents.
"Generate template" is unsupported for a nested structure like MAP and LIST type. Generate template is a link within the schema editor accessed through the Edit Schema button.
All expression Snap properties can be evaluated (when the '=' button is pressed) from pipeline parameters only, not from input documents from upstream Snaps. Input documents are data to be formatted and written to the target files.
The security model configured for the Groundplex (SIMPLE or KERBEROS authentication) must match the security model of the remote server. Due to the limitations of the Hadoop library, we can only create the necessary internal credentials to configure the Groundplex.
Parquet Snaps work well in a Linux environment. However, due to limitations in the Hadoop library on Windows, their functioning in a Windows environment may not always be as expected. We recommend you use a Linux environment for working with Parquet Snaps.
To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:
Download
hadoop.dll
andwinutils.exe
https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin (SnapLogic’s Hadoop version is 3.2.2)Create a temporary directory.
Place the
hadoop.dll
andwinutils.exe
files in this path:c:\winutils\bin
Set the environment variable HADOOP_HOME to point to
c:\winutils
Add
c:\winutils\bin
to the environment variable PATH as shown below:Add the JVM options in the Windows Snaplex:
jcc.jvm_options= -Djava.library.path=C:\\test
If you already have an existing
jvm_options
, then add:"-Djava.library.path=C:\\test"
after the space.
For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\test
Restart the JCC for configurations to take effect.
Known Issues
None.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document
|
| Mapper |
|
Output | Document |
| Mapper | A document with a filename for each Parquet file written. For example: |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:
Learn more about Error handling in Pipelines. |
Snap Settings
Asterisk ( * ): Indicates a mandatory field.
Suggestion icon (Emoji :Suggest_Icon: ): Indicates a list that is dynamically populated based on the configuration.
Expression icon (Emoji :Expression_Enabler: ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon ( ): Indicates that you can add fields in the field set.
Remove icon ( Emoji :Remove_icon: ): Indicates that you can remove fields from the field set.
Upload icon (Emoji :Upload_icon: ): Indicates that you can upload files.
Field Name | Field Type | Description |
---|---|---|
Label* Default Value: Parquet Writer | String | Specify a unique name for the Snap. |
Directory Default Value: hdfs://<hostname>:<port>/
| String/Expression/Suggestion | Specify the file path to a directory to read data from. The path must be in the following format: All files within the directory must be Parquet formatted. The following file storage systems are supported:
SnapLogic automatically appends The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property. |
File Filter Default Value: * | String/Expression | Specify the Glob file pattern. Use glob patterns to display a list of directories or files when you click the Suggestion icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property. |
File Default Value: N/A
| String/Expression/Suggestion | Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". |
Hive Metastore URL Default Value: N/A | String/Expression | Specify the URL of the Hive Metastore to assist in setting the schema along with the database and table setting. If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where you define the schema. |
Database Default value: N/A | String/Expression/Suggestion | Specify the Hive Metastore database which holds the schema for the outgoing data. |
Table Default value: N/A | String/Expression/Suggestion | Specify the table whose schema should be used for formatting the outgoing data. |
Fetch Hive Schema at Runtime Default value: Deselected | Checkbox | Select this checkbox to fetch the schema from the Metastore table before writing. The Snap fails to write if it cannot make connection to the metastore or the table does not exist during the Pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked. |
Edit Schema | Button | Specify a valid Parquet schema that describes the data. The schema can be specified based off a Hive Metastore table schema or generated from suggest data. Save the pipeline before editing the schema to generate suggest data that assists in specifying the schema based off of the schema of incoming documents. If no suggest data is available, then an example schema is generated along with documentation. Alter one of those schemas to describe the input data. The following is an example of a schema using all the primitive and some examples of logical types: message document { # Primitive Types optional int64 32_num; optional int64 64_num; optional boolean truth; optional binary message; optional float pi; optional double e; optional int96 96_num; optional fixed_len_byte_array (1) one_byte; # Logical Types optional binary snowman (UTF8); optional int32 8_num (INT_8); optional int32 16_num (INT_16); optional int32 u8_num (UINT_8); optional int32 u16_num (UINT_16); optional int32 u32_num (UINT_32); optional int64 u64_num (UINT_64); optional int32 dec_num (DECIMAL(5,2)); optional int32 jan7 (DATE); optional int32 noon (TIME_MILLIS); optional int64 jan7_epoch (TIMESTAMP_MILLIS); optional binary embedded (JSON); } "Generate template" will not work for nested structure like MAP and LIST type. |
Compression* | Dropdown list | Select the type of compression to use when writing the file. The available options are:
|
Partition by Default Value: N/A | String/Suggestion | Specify or select the key which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Refer to the 'Partition by' example below for an illustration. |
Azure SAS URI Properties | Shared Access Signatures (SAS) properties of the Azure Storage account. | |
SAS URI | String/Expression | Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account. You can get a valid SAS URI either from the Shared access signature in the Azure Portal or by generating from the SAS Generator Snap. If SAS URI value is provided in the Snap settings, then the account settings (in case any account is attached) are ignored. |
Decimal Rounding Mode
Default Value: Half up Example: Up | Dropdown list | Select the required rounding method for decimal values when they exceed the required number of decimal places. The available options are:
|
Snap Execution | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Troubleshooting
Error | Reason | Resolution |
---|---|---|
Unable to connect to the Hive Metastore. | This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore. | Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:
|
Parquet Snaps may not work as expected in Windows environment. | Because of the limitations in the Hadoop library on Windows, the functioning of Parquet Snaps is not as expected. | To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:
|
Additional Information
Write to S3 files with HDFS version CDH 5.8 or later
When running the HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:
Go to HDFS configuration.
In Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, add an entry with the following details:
Name: fs.s3a.threads.max
Value: 15
Click Save.
Restart all the nodes.
Under Restart Stale Services, select Re-deploy client configuration.
Click Restart Now.
{ "error": "Add-on 'biz.artemissoftware.confluence.multiexcerpt.MultiExcerptMacro' disallowed to impersonate the user because 'no valid active user exists'" }
If you're experiencing issues please see our Troubleshooting Guide.
Examples
Parquet Writer Snap with the Second Input View
The following example pipeline demonstrates the usage of the second input view to receive the table metadata. You can provide the Hive Metastore information in the Hive Metastore URL field, wherein a single input view is sufficient.
When you enable the second input view, the Snap overrides other schema settings, such as the schema within the Edit Schema box or the Hive Metastore-related properties and only accepts the schema from the second input view.
When you disable the second input view, the Snap receives the Schema from the Hive Metastore URL field.
The Parquet Writer Snap configuration with Directory path: | The Parquet Writer Snap View: |
The Hive Execute Snap with the table metadata information to pass to the second input view of the Parquet Writer Snap: | The Hive Execute Snap Output: |
Inserting and Querying Custom Metadata from the Flight Metadata Table
The Pipeline in this zipped example, MetadataCatalog_Insert_Read_Example.zip, demonstrates how you can:
Use the Catalog Insert Snap to update metadata tables.
Use the Catalog Query Snap to read the updated metadata information.
In this example:
We import a file containing the metadata.
We create a parquet file using the data in the imported file
We insert metadata that meets specific requirements into a partition in the target table.
We read the newly-inserted metadata using the Catalog Query Snap.