Content Comparison

...

Download hadoop.dlland winutils.exe https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin (SnapLogic’s Hadoop version is 3.2.2)
Create a temporary directory.
Place the hadoop.dlland winutils.exe files in this path: C:\\hadoop\bin
Set the environment variable HADOOP_HOME to point to C:\\hadoop
Add C:\hadoop\bin to the environment variable PATH as shown below:
Add the JVM options in the Windows Snaplex:jcc.jvm_options= -Djava.library.path=C:\\hadoop\bin
If you already have an existing jvm_options, then add: "-Djava.library.path=C:\\hadoop\bin" after the space.
For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\hadoop\bin
Restart the JCC for configurations to take effect.

...

Snap Execution

Field Name

Field Type

Description

Label*

Default Value: Parquet Writer
Example: Parquet Writer

String

Specify a unique name for the Snap.

Directory

Default Value: hdfs://<hostname>:<port>/
Example:

hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/
webhdfs://cdh-ga-2.fullsail.yourcompany.com:50070/user/ec2-user/csv/
_dirname

String/Expression/Suggestion

Specify the file path to a directory to read data from. The path must be in the following format:
hdfs://<hostname>:<port>/

All files within the directory must be Parquet formatted. The following file storage systems are supported:

hdfs
Protocol: hdfs://<hostname>:<port>/<path to directory>
Example: hdfs://localhost:8020/tmp
S3
Protocol: S3://<testbucket>/<key name prefix>
Example: S3://test-bucker/tmp
wasb
Protocol: wasb:///<storage container>/path to directory>
Example: wasb:///container/tmp
wasbs
Protocol: wasbs:///<storage container>/path to directory>
Example: wasbs:///container/tmp
adl
Protocol: adl://<store name>/<path to directory>
Example: adl://storename/tmp
adls
Protocol: adls://<store name>/<path to directory>
Example: adls://storename/tmp
abfs
Protocol: abfs:///filesystem/<path>/
Example: abfs://filesystem1/core.windows.net/dirl
abfs
Protocol: abfs://<filesystem>@<accountname>.<endpoint>/<path>
Example: abfs://filesystem2@adlsgen2test1.dfs.core.windows.net/dirl
abfss
Protocol: abfss:///filesystem/<path>/
Example: abfss://filesystem1/core.windows.net/dirl
abfss
Protocol: abfss://<filesystem>@<accountname>.<endpoint>/<path>
Example: abfss://filesystem2@adlsgen2test1.dfs.core.windows.net/dirl

Note
SnapLogic automatically appends `azuredatalakestore.net` to the store name you specify when using Azure Data Lake; therefore, you do not have to add `azuredatalakestore.net` to the URI while specifying the directory.

The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.

File Filter

Default Value: *
Example: **

String/Expression

Specify the Glob file pattern.

Use glob patterns to display a list of directories or files when you click the Suggestion icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.

Expand

title	Glob Pattern Interpretation Rules

The following rules are used to interpret glob patterns:

- The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period.

- The ** characters match zero or more characters across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory.

- The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension.

- The \ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash, and \{ matches a left brace, for example.

- The ! character is used to exclude matching files from the output.

- The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'.

- Within a bracket expression, the '*', '?', and '\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating.

- The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'.

- Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name ".login".

- All other characters match themselves.

Examples:

'*.csv' matches all files with a csv extension in the current directory only.

'**.csv' matches all files with a csv extension in the current directory and in all its subdirectories.

*[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP.

File

Default Value: N/A
Example:

sample.csv
tmp/another.csv
_filename

String/Expression/Suggestion

Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/".
The File value can be a JavaScript expression which will be evaluated with values from the input view document. When you click the Suggest icon, the Snap displays a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.

Hive Metastore URL

Default Value: N/A
Example: thrift://localhost:9083

String/Expression

Specify the URL of the Hive Metastore to assist in setting the schema along with the database and table setting. If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where you define the schema.

Database

Default value: N/A

String/Expression/Suggestion

Specify the Hive Metastore database which holds the schema for the outgoing data.

Table

Default value: N/A

String/Expression/Suggestion

Specify the table whose schema should be used for formatting the outgoing data.

Fetch Hive Schema at Runtime

Default value: Deselected

Checkbox

Select this checkbox to fetch the schema from the Metastore table before writing. The Snap fails to write if it cannot make connection to the metastore or the table does not exist during the Pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked.

Edit Schema

Button

Specify a valid Parquet schema that describes the data.

The schema can be specified based off a Hive Metastore table schema or generated from suggest data. Save the pipeline before editing the schema to generate suggest data that assists in specifying the schema based off of the schema of incoming documents. If no suggest data is available, then an example schema is generated along with documentation. Alter one of those schemas to describe the input data.
The Parquet schema can also be written manually. A schema is defined by a list of fields and here is an example describing the contact information of a person.

Expand

After defining the message type, a list of fields are given. A field is comprised of a repetition, a type, and the field name. Available repetitions are required, optional, and repeated.
Each field has a type. The primitive types include:

binary -used for strings

fixed_len_byte_array - used for byte arrays of fixed length

boolean - a 1 bit boolean value

int32 - a 32 bit integer

int64 - a 64 bit integer

int96 - a 96 bit integer

float - a 32 bit floating point number

double - a 64 bit floating point number

These types can be annotated with a logical type to specify how the application should interpret the data. The Logical types include:

UTF8 - used with binary to specify the string as UTF8 encoded

INT_8 - used with int32 to specify the int as an 8 bit signed integer

INT_16 - used with int32 to specify the int as a 16 bit signed integer

Unsigned types - may be used to produce smaller in-memory representations of the data. If the stored value is larger than the maximum allowed by int32 or int64, then the behavior is undefined.

UINT_8 - used with int32 to specify the int as an 8 bit unsigned integer

UINT_16 - used with int32 to specify the int as a 16 bit unsigned integer

UINT_32 - used with int32 to specify the int as a 32 bit unsigned integer

UINT_64 - used with int64 to specify the int as a 64 bit unsigned integer

DECIMAL(precision, scale) - used to describe arbitrary-precision signed decimal numbers of the form value * 10^(-scale) to the given precision. The annotation can be with int32, int64, fixed_len_byte_array, binary. See the Parquet documentation for limits on precision that can be given.

DATE - used with int32 to specify the number of days since the Unix epoch, 1 January 1970

Note: This Snap supports only the following date format: yyyy-MM-dd.

TIME_MILLIS - used with int32 to specify the number of milliseconds after midnight

TIMESTAMP_MILLIS - used with int64 to store the number of milliseconds from the Unix epoch, 1 January 1970

INTERVAL - used with a fixed_len_byte_array of length 12, where the array stores 3 unsigned little-endian integers. These integers specify

a number in months

a number in days

a number in milliseconds

JSON - used with binary to represent an embedded JSON document

BSON - used for an embedded BSON document

The following is an example of a schema using all the primitive and some examples of logical types:

Code Block

message document {
  # Primitive Types
  optional int64 32_num;
  optional int64 64_num;
  optional boolean truth;
  optional binary message;
  optional float pi;
  optional double e;
  optional int96 96_num;
  optional fixed_len_byte_array (1) one_byte;

  # Logical Types
  optional binary snowman (UTF8);
  optional int32 8_num (INT_8);
  optional int32 16_num (INT_16);
  optional int32 u8_num (UINT_8);
  optional int32 u16_num (UINT_16);
  optional int32 u32_num (UINT_32);
  optional int64 u64_num (UINT_64);
  optional int32 dec_num (DECIMAL(5,2));
  optional int32 jan7 (DATE);
  optional int32 noon (TIME_MILLIS);
  optional int64 jan7_epoch (TIMESTAMP_MILLIS);
  optional binary embedded (JSON);
}

Note
"Generate template" will not work for nested structure like MAP and LIST type.

Compression*

Dropdown list

Select the type of compression to use when writing the file. The available options are:

NONE
SNAPPY
GZIP
LZO

To use LZO compression, you must explicitly enable the LZO compression type on the cluster (as an administrator) for the Snap to recognize and run the format. For more information, see Data Compression. For detailed guidance on setting up LZO compression, see Cloudera documentation on Installing the GPL Extras Parcel.
Many compression algorithms require both Java and system libraries and will fail if the latter is not installed. If you see unexpected errors, ask your system administrator to verify that all the required system libraries are installed–they are typically not installed by default. The system libraries will have names such as liblzo2.so.2 or libsnappy.so.1 and will probably be located in the /usr/lib/x86_64-linux-gnu directory.

Partition by

Default Value: N/A

String/Suggestion

Specify or select the key which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Refer to the 'Partition by' example below for an illustration.

Azure SAS URI Properties

Shared Access Signatures (SAS) properties of the Azure Storage account.

SAS URI

String/Expression

Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account.

You can get a valid SAS URI either from the Shared access signature in the Azure Portal or by generating from the SAS Generator Snap.

Note
If SAS URI value is provided in the Snap settings, then the account settings (in case any account is attached) are ignored.

Decimal Rounding Mode

Default Value: Half up

Example: Up

Dropdown list

Select the required rounding method for decimal values when they exceed the required number of decimal places. The available options are:

Half up
Half down
Half even
Up
Down
Ceiling
Floor
Truncate

Timestamp Parquet type

Dropdown list

Select one of the following three modes in which the Snap executes:

Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

Execute

Choose the appropriate Parquet type for your timestamp schema based on the format of the timestamp data. This schema is passed from the second input view. The available options are:

INT96
INT64 timestamp millis
INT64 timestamp micros
INT64 timestamp nanos

If the timestamp data is in string format, a date-time object, or milliseconds, select one of the INT64 types (Millis, Micros, or Nanos) from the dropdown.

When reading timestamp data from the Parquet Reader Snap, select the Use datetime types checkbox.
If the timestamp is represented as a 12-byte INT96 timestamp (12-byte represented timestamp), you must select the INT96 option. When reading from the Parquet Reader Snap, select the INT96 as the timestamp option.

Default value: INT96
Example: INT64 Timestamp millis

Snap Execution

Dropdown list

Select one of the following three modes in which the Snap executes:

Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
Disabled: Disables the Snap and all Snaps that are downstream from it.

...

Error

Reason

Resolution

Unable to connect to the Hive Metastore.

This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore.

Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:

Enable the 'Schema View' in the Parquet Writer Snap by adding the second Input View.
1. Connect a Hive Execute Snap to the Schema View. Configure the Hive Execute Snap to execute the DESCRIBE TABLE command to read the table metadata and feed it to the schema view.

Parquet Snaps may not work as expected in the Windows environment.

Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected.

To use the Parquet Writer Snap on a Windows Snaplex, follow these steps:

Create a temporary directory. For example: C:\test\hadoop\.
Place two files, "hadoop.dll" and "winutils.exe", in the newly created temporary directory. Use this link https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin to download hadoop.dll and winutills.exe. (SnapLogic’s code base Hadoop is 3.2.2).
Add the JVM options in the Windows Plex as shown below:
jcc.jvm_options = -Djava.library.path=C:\\testhadoop
If you already have existing jvm_options, then add the following "-Djava.library.path=C:\\testhadoop" after the space. For example:
jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\testhadoop
Restart the JCC for configurations to take effect.

Failure: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)',

Because of the limitations in the Hadoop library on Windows, Parquet Snaps does not function as expected.

To resolve this issue, follow these steps:

Download hadoop.dlland winutils.exe https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin (SnapLogic’s Hadoop version is 3.2.2)
Create a temporary directory.
Place the hadoop.dlland winutils.exe files in this path: C:\\hadoop\bin
Set the environment variable HADOOP_HOME to point to C:\\hadoop
Add C:\hadoop\bin to the environment variable PATH as shown below:
Variable name: PATH
Variable value: VEN_HOME%\bin;%HADOOP_HOME%\bin
Add the JVM options in the Windows Snaplex:jcc.jvm_options= -Djava.library.path=C:\\hadoop\bin
If you already have an existing jvm_options, then add: "-Djava.library.path=C:\\hadoop\bin" after the space.
For example:jcc.jvm_options = -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 -Djava.library.path=C:\\\hadoop\bin
Restart the JCC for configurations to take effect.