Input 

Document

• Min: 1

• Max: 2

Mapper

• This Snap has one or two document input views. When you enable the second input view, the Snap ignores other schema settings such as Schema button or Hive Metastore related properties, but it accepts the schema from the second input view only. However, when you disable the second input view, the Snap prepares to receive the Schema with the provided information on the Hive Metastore URL property. The supported data types are:

  • Primitive: Boolean, Integer, Float, double, and byte-array

  • Local: map, list

• The Snap expects a Hive Execute Snap that contains the "Describe table" statement in the second input view.

Output

Document

• Min: 0

• Max: 1

Mapper

A document with a filename for each Parquet file written. For example: {"filename" : "hdfs://localhost/tmp/2017/april/sample.parquet"}

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

• Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

• Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

• Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

String

Specify a unique name for the Snap.

String/Expression/Suggestion

Specify the file path to a directory to read data from. The path must be in the following format:
hdfs://<hostname>:<port>/

All files within the directory must be Parquet formatted. The following file storage systems are supported:

• hdfs
  Protocol: hdfs://<hostname>:<port>/<path to directory>
  Example: hdfs://localhost:8020/tmp

• S3
  Protocol: S3://<testbucket>/<key name prefix>
  Example: S3://test-bucker/tmp

• wasb
  Protocol: wasb:///<storage container>/path to directory>
  Example: wasb:///container/tmp

• wasbs
  Protocol: wasbs:///<storage container>/path to directory>
  Example: wasbs:///container/tmp

• adl
  Protocol: adl://<store name>/<path to directory>
  Example: adl://storename/tmp

• adls
  Protocol: adls://<store name>/<path to directory>
  Example: adls://storename/tmp

• abfs
  Protocol: abfs:///filesystem/<path>/
  Example: abfs://filesystem1/core.windows.net/dirl

• abfs
  Protocol: abfs://<filesystem>@<accountname>.<endpoint>/<path>
  Example: abfs://filesystem2@adlsgen2test1.dfs.core.windows.net/dirl

• abfss
  Protocol: abfss:///filesystem/<path>/
  Example: abfss://filesystem1/core.windows.net/dirl

• abfss
  Protocol: abfss://<filesystem>@<accountname>.<endpoint>/<path>
  Example: abfss://filesystem2@adlsgen2test1.dfs.core.windows.net/dirl

The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.

String/Expression

Specify the Glob file pattern.

String/Expression/Suggestion

Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". 
The File value can be a JavaScript expression which will be evaluated with values from the input view document. When you click the Suggest icon, the Snap displays a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.

String/Expression

Specify the URL of the Hive Metastore to assist in setting the schema along with the database and table setting. If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where you define the schema. 

String/Expression/Suggestion

Specify the Hive Metastore database which holds the schema for the outgoing data.

String/Expression/Suggestion

Specify the table whose schema should be used for formatting the outgoing data.

Checkbox

Select this checkbox to fetch the schema from the Metastore table before writing. The Snap fails to write if it cannot make connection to the metastore or the table does not exist during the Pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked.

Button

Specify a valid Parquet schema that describes the data.  

The following is an example of a schema using all the primitive and some examples of logical types:

message document {
  # Primitive Types
  optional int64 32_num;
  optional int64 64_num;
  optional boolean truth;
  optional binary message;
  optional float pi;
  optional double e;
  optional int96 96_num;
  optional fixed_len_byte_array (1) one_byte;

  # Logical Types
  optional binary snowman (UTF8);
  optional int32 8_num (INT_8);
  optional int32 16_num (INT_16);
  optional int32 u8_num (UINT_8);
  optional int32 u16_num (UINT_16);
  optional int32 u32_num (UINT_32);
  optional int64 u64_num (UINT_64);
  optional int32 dec_num (DECIMAL(5,2));
  optional int32 jan7 (DATE);
  optional int32 noon (TIME_MILLIS);
  optional int64 jan7_epoch (TIMESTAMP_MILLIS);
  optional binary embedded (JSON);
}

Dropdown list

Select the type of compression to use when writing the file. The available options are:

• NONE

• SNAPPY

• GZIP

• LZO

String/Suggestion

Specify or select the key which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Refer to the 'Partition by' example below for an illustration. 

Shared Access Signatures (SAS) properties of the Azure Storage account.

String/Expression

Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account.

Dropdown list

Select the required rounding method for decimal values when they exceed the required number of decimal places. The available options are:

• Half up

• Half down

• Half even

• Up

• Down

• Ceiling

• Floor

• Truncate

Dropdown list

Select one of the following three modes in which the Snap executes:

• Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

• Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

• Disabled: Disables the Snap and all Snaps that are downstream from it.

Unable to connect to the Hive Metastore.

This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore.

Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:

1. Enable the 'Schema View' in the Parquet Writer Snap by adding the second Input View.

   1. Connect a Hive Execute Snap to the Schema View. Configure the Hive Execute Snap to execute the DESCRIBE TABLE command to read the table metadata and feed it to the schema view. 

Parquet Snaps may not work as expected in the Windows environment.

Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected.

To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:

1. Create a temporary directory. For example: C:\test\.

2. Place two files, "hadoop.dll" and "winutils.exe", in the newly created temporary directory. Use this link ​https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin to download hadoop.dll and winutills.exe. (SnapLogic’s code base Hadoop is 3.2.2).

3. Add the JVM options in the Windows Plex as shown below:
   jcc.jvm_options = -Djava.library.path=C:\\test

4. If you already have existing jvm_options, then add the following "-Djava.library.path=C:\\test" after the space. For example:
   jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\test

5. Restart the JCC for configurations to take effect.

Failure: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)',

Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected.

To resolve this issue, follow these steps:

1. Download hadoop.dlland winutils.exe ​https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin (SnapLogic’s Hadoop version is 3.2.2)

2. Create a temporary directory.

3. Place the hadoop.dlland winutils.exe files in this path: C:\hadoop\bin

4. Set the environment variable HADOOP_HOME to point to C:\hadoop

5. Add C:\hadoop\bin to the environment variable PATH as shown below:
   Variable name: PATH
   Variable value: VEN_HOME%\bin;%HADOOP_HOME%\bin

6. Add the JVM options in the Windows Snaplex:jcc.jvm_options= -Djava.library.path=C:\hadoop\bin

   If you already have an existing jvm_options, then add: "-Djava.library.path=C:\hadoop\bin" after the space.
   For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\hadoop\bin

7. Restart the JCC for configurations to take effect.

The Parquet Writer Snap configuration with Directory path:

The Parquet Writer Snap View:

The Hive Execute Snap with the table metadata information to pass to the second input view of the Parquet Writer Snap:

The Hive Execute Snap Output: