...
You can use this Snap to convert documents to the Parquet format and write the data to HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Blob File Storage Gen 2), WASB (Azure storage), or an S3 bucket. This Snap supports a nested schema such as LIST and MAP. You can also use this Snap to write schema information to the Catalog Insert Snap.
This Snap supports HDFS, ADL (Azure Data Lake), ABFS (Azure Data Lake Blob File Storage Gen 2), WASB (Azure storage), and S3 protocols.
...
Auto schema generation in this Snap involves the exclusion of excludes null fields. For example, if the Snap receives ten input documents during preview execution, and four of these documents contain null values for certain fields in all instances, those four fields are disregarded during schema generation. The schema only includes fields with at least one non-null value among the preview input documents.
"Generate template" is unsupported for a nested structure like MAP and LIST type. Generate template is a link within the schema editor accessed through the Edit Schema button.
All expression Snap properties can be evaluated (when the '=' button is pressed) from pipeline parameters only, not from input documents from upstream Snaps. Input documents are data to be formatted and written to the target files.
The security model configured for the Groundplex (SIMPLE or KERBEROS authentication) must match the security model of the remote server. Due to the limitations of the Hadoop library, we can only create the necessary internal credentials to configure the Groundplex.
Parquet Snaps work well in a Linux environment. However, due to limitations in the Hadoop library on Windows, their functioning in a Windows environment may not always be as expected. We recommend you use a Linux environment for working with Parquet Snaps.
...
Download
hadoop.dll
andwinutils.exe
https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin (SnapLogic’s Hadoop version is 3.2.2)Create a temporary directory.
Place the
hadoop.dll
andwinutils.exe
files in this path:cC:\winutils\hadoop\bin
Set the environment variable
HADOOP_HOME
to point tocC:\winutils\hadoop
Add
cC:\winutilshadoop\bin
to the environment variable PATH as shown below:Add the JVM options in the Windows Snaplex:
jcc.jvm_options= -Djava.library.path=C:\\testhadoop\bin
If you already have an existing
jvm_options
, then add:"-Djava.library.path=C:\\testhadoop\bin"
after the space.
For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\hadoop\testbin
Restart the JCC for configurations to take effect.
Known Issues
None.
Snap Views
...
Type
...
Format
...
Number of Views
Multiexcerpt include macro | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Learn more about the Azure Storage library upgrade.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document
|
| Mapper |
|
Output | Document |
| Mapper | A document with a filename for each Parquet file written. For example: |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:
Learn more about Error handling in Pipelines. |
Snap Settings
Info |
---|
|
Field Name | Field Type | Description | ||||||
---|---|---|---|---|---|---|---|---|
Label* Default Value: Parquet Writer | String | Specify a unique name for the Snap. | ||||||
Directory Default Value: hdfs://<hostname>:<port>/
| String/Expression/Suggestion | Specify the file path to a directory to read data from. The path must be in the following format: All files within the directory must be Parquet formatted. The following file storage systems are supported:
The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property. | ||||||
File Filter Default Value: * | String/Expression | Specify the Glob file pattern. Use glob patterns to display a list of directories or files when you click the Suggestion icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.
| ||||||
File Default Value: N/A
| String/Expression/Suggestion | Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". | ||||||
Hive Metastore URL Default Value: N/A | String/Expression | Specify the URL of the Hive Metastore to assist in setting the schema along with the database and table setting. If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where you define the schema. | ||||||
Database Default value: N/A | String/Expression/Suggestion | Specify the Hive Metastore database which holds the schema for the outgoing data. | ||||||
Table Default value: N/A | String/Expression/Suggestion | Specify the table whose schema should be used for formatting the outgoing data. | ||||||
Fetch Hive Schema at Runtime Default value: Deselected | Checkbox | Select this checkbox to fetch the schema from the Metastore table before writing. The Snap fails to write if it cannot make connection to the metastore or the table does not exist during the Pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked. | ||||||
Edit Schema | Button | Specify a valid Parquet schema that describes the data. The schema can be specified based off a Hive Metastore table schema or generated from suggest data. Save the pipeline before editing the schema to generate suggest data that assists in specifying the schema based off of the schema of incoming documents. If no suggest data is available, then an example schema is generated along with documentation. Alter one of those schemas to describe the input data.
The following is an example of a schema using all the primitive and some examples of logical types:
| ||||||
Compression* | Dropdown list | Select the type of compression to use when writing the file. The available options are:
| ||||||
Partition by Default Value: N/A | String/Suggestion | Specify or select the key which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Refer to the 'Partition by' example below for an illustration. | ||||||
Azure SAS URI Properties | Shared Access Signatures (SAS) properties of the Azure Storage account. | |||||||
SAS URI | String/Expression | Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account. You can get a valid SAS URI either from the Shared access signature in the Azure Portal or by generating from the SAS Generator Snap.
| ||||||
Decimal Rounding Mode
Default Value: Half up Example: Up | Dropdown list | Select the required rounding method for decimal values when they exceed the required number of decimal places. The available options are:
| ||||||
Snap Execution | Dropdown list | Select one of the following three modes in which the Snap executes:
|
Troubleshooting
Error | Reason | Resolution |
---|---|---|
Unable to connect to the Hive Metastore. | This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore. | Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:
|
Parquet Snaps may not work as expected in the Windows environment. | Because of the limitations in the Hadoop library on Windows, the functioning of Parquet Snaps is does not function as expected. | To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:
|
Additional Information
...
| Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected. | To resolve this issue, follow these steps:
|
...
Additional Information
Write to S3 files with HDFS version CDH 5.8 or later
When running the HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:
...
The following example pipeline demonstrates the usage of the second input view to receive the table metadata. You can provide the Hive Metastore information in the Hive Metastore URL field, wherein a single input view is sufficient.
...
When you enable the second input view, the Snap overrides other schema settings, such as the schema within the Edit Schema box or the Hive Metastore-related properties and only accepts the schema from the second input view.
When you disable the second input view, the Snap receives the Schema from the Hive Metastore URL field.
The Parquet Writer Snap configuration with Directory path: | The Parquet Writer Snap View: |
The Hive Execute Snap with the table metadata information to pass to the second input view of the Parquet Writer Snap: | The Hive Execute Snap Output: |
Expand | ||
---|---|---|
| ||
Here is an example of a Parquet Writer configured to write to a local instance of HDFS. The output is written to /tmp/parquet-example. There is no Hive Metastore configured and no compression used. See the documentation on the Schema setting to view an example of the schema. |
Expand | ||
---|---|---|
| ||
Example on "Partition by":Assume the following input documents: [ The settings of the Parquet Writer Snap are as follows: The pipeline execution will generate two files:
The key-value pairs for "month" and "day" will not be included in the output files. |
Expand | ||
---|---|---|
| ||
Reading and writing Parquet files from/to AWS S3 requires an S3 account.
|
...
Expand | ||
---|---|---|
| ||
Example_Parquet_Writer_with Second_Input_View.slp Sept 11, 2023 by Gouri Bhagchandani |
Snap Pack History
...
Downloads
Info |
---|
|
Attachments | ||
---|---|---|
|
Snap Pack History
Insert excerpt | ||||||||
---|---|---|---|---|---|---|---|---|
|