Parquet Writer

On this Page

Snap Type:

Write

Description:

This Snap converts documents into the Parquet format and writes the data to HDFS or S3. A nested schema such as LIST and MAP are also supported by the Snap. You can also use this Snap to write schema information into the Catalog Insert Snap.

  • Expected upstream Snaps: Any Snap with a document output view. The Snap expects a Hive Execute Snap that contains the "Describe table" statement in the second input view.
  • Expected downstream Snaps: Any Snap with a document input view
  • Expected input: A document.
  • Expected output: A document with a filename for each Parquet file written.
    Example: {"filename" : "hdfs://localhost/tmp/2017/april/sample.parquet"}

 Modes

Prerequisites:

The user must have access and permission to write to HDFS or AWS S3. 

Limitations and Known Issues:
  • "Generate template" will not work for a nested structure like MAP and LIST type. Generate template is a button within the schema editor accessed through the Edit Schema property.
  • All expression Snap properties can be evaluated (when the '=' button is pressed) from pipeline parameters only, not from input documents. Input documents are data to be formatted and written to the target files. 
  • Parquet Snaps work well in a Linux environment. However, due to limitations in the Hadoop library on Windows, their functioning in a Windows environment may not always be as expected. We recommend you use a Linux environment for working with Parquet Snaps.
 How to Use the Parquet Writer Snap on a Windows Plex

The Parquet Writer Snap is tested against Windows Server 2008, 2010 and 2012.

To use the Parquet Writer Snap on a Windows Plex:

  1. Create a temporary directory. For example: C:\test\.
  2. Place two files, "hadoop.dll" and "winutils.exe", in the newly created temporary directory.
  3. Add the JVM options in the Windows Plex as shown below:

  4. If you already have existing jvm_options, then add the following "-Djava.library.path=C:\\test" after the space. For example:

  5. Restart the JCC for configurations to take effect.

You cannot use a SAS URI (generated on a specific blob) through the SAS Generator Snap.

Account: 

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed below.

The security model configured for the groundplex (SIMPLE or KERBEROS authentication) must match the security model of the remote server. Due to limitations of the Hadoop library we are only able to create the necessary internal credentials for the configuration of the groundplex.

Views:
Input

This Snap has one or two document input views. When the second input view is enabled, the Snap ignores other schema settings like schema button or Hive Metastore related properties, but it accepts the schema from the second input view only. However, when the second input view is disabled, the Snap prepares to receive the Schema with the provided information on the Hive Metastore URL property.

Supported data types: 

  • Primitive: boolean, integer, float, double, and byte_array
  • Logical: map, list
Output

This Snap has at most one document output view.

ErrorThis Snap has at most one document error view and produces zero or more documents in the view.

Settings

Label

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Directory



A directory in the supported file storage systems to read data. All files within the directory must be Parquet formatted.

We support file storage systems as follows:

Protocol
Directory Format
Example
hdfshdfs://<hostname>:<port>/hdfs://localhost:8020/tmp
s3s3://<testbucket>/<folder>s3://test-bucket/tmp
wasbwasb:///<storage container>/path to directory>wasb:///container/tmp
wasbswasbs:///<storage container>/path to directory>wasbs:///container/tmp
adladl://<store name>/<path to directory>adl://storename/tmp
adlsadls://<store name>/<path to directory>adls://storename/tmp
abfs

abfs:///filesystem/<path>/

abfs://filesystem@accountname.endpoint/<path>

abfs:///filesystem2/dir1

abfs://filesystem2@snaplogicaccount.dfs.core.windows.net/dir1

SnapLogic automatically appends azuredatalakestore.net to the store name you specify when using Azure Data Lake; therefore, you do not need to add azuredatalakestore.net to the URI while specifying the directory.

The Directory property is not used in the pipeline execution or preview and used only in the Suggest operation. When you click on the Suggest icon, it will display a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.


Example

  • hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/
  • webhdfs://cdh-ga-2.fullsail.yourcompany.com:50070/user/ec2-user/csv/
  • _dirname

Default value:  hdfs://<hostname>:<port>/

Filter


Use glob patterns to display a list of directories or files when you click the Suggest icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.

Default Value: *

For more information on glob patterns, click the link below.


 Glob Pattern Interpretation Rules

Glob Pattern Interpretation Rules

The following rules are used to interpret glob patterns:

  • The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period.

  • The ** characters match zero or more characters across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory.

  • The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension.

  • The \ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash, and \{ matches a left brace, for example.

  • The ! character is used to exclude matching files from the output. 
  • The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'.

    Within a bracket expression, the '*', '?', and '\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating.

  • The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'.

  • Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name ".login".

  • All other characters match themselves.

Examples:

  • '*.csv' matches all files with a csv extension in the current directory only.
  • '**.csv' matches all files with a csv extension in the current directory and in all its subdirectories.
  • *[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP.

File




Filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". The File property can be a JavaScript expression which will be evaluated with values from the input view document. When you press the Suggest icon, it will display a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.

Example: 

  • sample.csv
  • tmp/another.csv
  • _filename

Default value:  [None]

User Impersonation

Select this check box to enable user impersonation.

For encryption zones, use user impersonation. 

Default value:  Not selected

For more information on working with user impersonation, click the link below.


 User Impersonation Details

Generic User Impersonation Behavior

When the User Impersonation check box is selected, and Kerberos is the account type, the Client Principal configured in the Kerberos account impersonates the pipeline user.

When the User Impersonation option is selected, and Kerberos is not the account type, the user executing the pipeline is impersonated to perform HDFS Operations. For example, if the user logged into the SnapLogic platform is operator@snaplogic.com, the user name "operator" is used to proxy the super user. 

User impersonation behavior on pipelines running on Groundplex with a Kerberos account configured in the Snap

  • When the User Impersonation checkbox is selected in the Snap, it is the pipeline user who performs the file operation. For example, if the user logged into the SnapLogic platform is operator@snaplogic.com, the user name "operator" is used to proxy the super user.
  • When the User Impersonation checkbox is not selected in the Snap, the Client Principal configured in the Kerberos account performs the file operation.



For non-Kerberised clusters, you must activate Superuser access in the Configuration settings.


HDFS Snaps support the following accounts:

  • Azure storage account
  • Azure Data Lake account
  • Kerberos account
  • No account

When an account is configured with an HDFS Snap, user impersonation settings have no impact on all accounts, except the Kerberos account.

Hive Metastore URL


This setting is used to assist in setting the schema along with the database and table setting.  If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it.  Set the value to a Hive Metastore url where the schema is defined. 

Default value: [None]

Database


The Hive Metastore database where the schema is defined. See the Hive Metastore URL setting for more information.

Default value: [None]

Table

The table to read the schema from in the Hive Metastore database. See the Hive Metastore URL setting for more information.

Default value: [None]

Fetch Hive Schema at Runtime

When set, will fetch the schema from the Metastore table before writing. Will fail to write if cannot make connection to the metastore or the table does not exist during the pipeline's execution. Will use the metastore schema instead of the one set in the Snap's Edit Schema property if this is checked.

Default value: Not selected

Edit Schema


A valid Parquet schema that describes the data.  The schema can be specified based off a Hive Metastore table schema or generated from suggest data.  Save the pipeline before editing the schema to generate suggest data that will assist in specifying the schema based off of the schema of incoming documents.  If no suggest data is available, then some documentation and an example schema will be generated instead.  Alter one of those schemas to describe the input data.

The Parquet schema can also be written manually.  A schema is defined by a list of fields and here is an example describing the contact information of a person. 

After defining the message type, a list of fields are given.  A field is comprised of a repetition, a type, and the field name. Available repetitions are required, optional, and repeated.

Each field has a type. The primitive types include:

  • binary - used for strings
  • fixed_len_byte_array - used for byte arrays of fixed length
  • boolean - a 1 bit boolean value
  • int32 - a 32 bit integer
  • int64 - a 64 bit integer
  • int96 - a 96 bit integer
  • float - a 32 bit floating point number
  • double - a 64 bit floating point number

These types can be annotated with a logical type to specify how the application should interpret the data. The Logical types include:
  • UTF8 - used with binary to specify the string as UTF8 encoded
  • INT_8 - used with int32 to specify the int as an 8 bit signed integer
  • INT_16 - used with int32 to specify the int as a 16 bit signed integer
  • Unsigned types - may be used to produce smaller in-memory representations of the data. If the stored value is larger than the maximum allowed by int32 or int64, then the behavior is undefined.
    • UINT_8 - used with int32 to specify the int as an 8 bit unsigned integer
    • UINT_16 - used with int32 to specify the int as a 16 bit unsigned integer
    • UINT_32 - used with int32 to specify the int as a 32 bit unsigned integer
    • UINT_64 - used with int64 to specify the int as a 64 bit unsigned integer
  • DECIMAL(precisionscale) - used to describe arbitrary-precision signed decimal numbers of the form value * 10^(-scale) to the given precision. The annotation can be with int32, int64, fixed_len_byte_array, binary. See the Parquet documentation for limits on precision that can be given.
  • DATE - used with int32 to specify the number of days since the Unix epoch, 1 January 1970

    This Snap supports only the following date format: yyyy-MM-dd.

  • TIME_MILLIS - used with int32 to specify the number of milliseconds after midnight
  • TIMESTAMP_MILLIS - used with int64 to store the number of milliseconds from the Unix epoch, 1 January 1970
  • INTERVAL - used with a fixed_len_byte_array of length 12, where the array stores 3 unsigned little-endian integers. These integers specify
    • a number in months
    • a number in days
    • a number in milliseconds
  • JSON - used with binary to represent an embedded JSON document
  • BSON - used for an embedded BSON document
The following is an example of a schema using all the primitive and some examples of logical types:
message document {
  # Primitive Types
  optional int64 32_num;
  optional int64 64_num;
  optional boolean truth;
  optional binary message;
  optional float pi;
  optional double e;
  optional int96 96_num;
  optional fixed_len_byte_array (1) one_byte;

  # Logical Types
  optional binary snowman (UTF8);
  optional int32 8_num (INT_8);
  optional int32 16_num (INT_16);
  optional int32 u8_num (UINT_8);
  optional int32 u16_num (UINT_16);
  optional int32 u32_num (UINT_32);
  optional int64 u64_num (UINT_64);
  optional int32 dec_num (DECIMAL(5,2));
  optional int32 jan7 (DATE);
  optional int32 noon (TIME_MILLIS);
  optional int64 jan7_epoch (TIMESTAMP_MILLIS);
  optional binary embedded (JSON);
}

"Generate template" will not work for nested structure like MAP and LIST type.

Compression


Required. The type of compression to use when writing the file. The available options are:

  • NONE
  • SNAPPY
  • GZIP
  • LZO
  • To use LZO compression, you must explicitly enable the LZO compression type on the cluster (as an administrator) for the Snap to recognize and run the format. For more information, see Data Compression. For detailed guidance on setting up LZO compression, see Cloudera documentation on Installing the GPL Extras Parcel.
  • Many compression algorithms require both Java and system libraries and will fail if the latter is not installed. If you see unexpected errors, ask your system administrator to verify that all the required system libraries are installed–they are typically not installed by default. The system libraries will have names such as liblzo2.so.2 or libsnappy.so.1 and will probably be located in the /usr/lib/x86_64-linux-gnu directory.
Partition by

Press '+' button to add a new row and use the suggest button to select a key name in the input document, which will be used to get the 'Partition by' folder name. All input documents should contain this key name or an error document will be written to the error view. Please see the 'Partition by' example below for an illustration. 

Default value: [None]

Azure SAS URI PropertiesShared Access Signatures (SAS) properties of the Azure Storage account.
SAS URI

Specify the Shared Access Signatures (SAS) URI that you want to use to access the Azure storage blob folder specified in the Azure Storage Account.

You can get a valid SAS URI either from the Shared access signature in the Azure Portal or by generating from the SAS Generator Snap.

If SAS URI value is provided in the Snap settings, then the account settings (in case any account is attached) are ignored.

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Troubleshooting

  • To generate a schema based on suggest data, the Pipeline must have the suggest-data available in the browser. This may require the user to save the Pipeline before editing the schema.
  • The Snap can only write data into HDFS.

Writing to S3 files with HDFS version CDH 5.8 or later

When running HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:

  1. Go to HDFS configuration.
  2. In Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, add an entry with the following details:
    • Name: fs.s3a.threads.max
    • Value: 15
  3. Click Save.
  4. Restart all the nodes.
  5. Under Restart Stale Services, select Re-deploy client configuration.
  6. Click Restart Now.

Unable to Connect to the Hive Metastore

Error Message: Unable to connect to the Hive Metastore.

Description: This error occurs when the Parquet Writer Snap is unable to fetch schema for Kerberos-enabled Hive Metastore.

Resolution: Pass the Hive Metastore's schema directly to the Parquet Writer Snap. To do so:

  1. Enable the 'Schema View' in the Parquet Writer Snap by adding the second Input View.
  2. Connect a Hive Execute Snap to the Schema View. Configure the Hive Execute Snap to execute the DESCRIBE TABLE command to read the table metadata and feed it to the schema view. 

Temporary Files

During execution, data processing on Snaplex nodes occurs principally in-memory as streaming and is unencrypted. When larger datasets are processed that exceeds the available compute memory, the Snap writes Pipeline data to local storage as unencrypted to optimize the performance. These temporary files are deleted when the Snap/Pipeline execution completes. You can configure the temporary data's location in the Global properties table of the Snaplex's node properties, which can also help avoid Pipeline errors due to the unavailability of space. For more information, see Temporary Folder in Configuration Options

Examples


 Parquet Writer with the second input view

In the below pipeline, the Parquet Writer Snap receives metadata of the table from the second input view. The Hive Metastore information may be directly provided into the Snap using the Hive Metastore URL property wherein a single input view is sufficient. 

When the second input view is enabled, the Snap ignores other schema settings like schema button or Hive Metastore related properties, but it accepts the schema from the second input view only. However, when the second input view is disabled, the Snap prepares to receive the Schema with the provided information on the Hive Metastore URL property.

 

The Parquet Writer Snap with Directory path:

The Hive Execute Snap with the table metadata information from the second input view of the Parquet Writer Snap:

 Parquet Writer configured to write to a local instance of HDFS

Here is an example of a Parquet Writer configured to write to a local instance of HDFS. The output is written to /tmp/parquet-example.  There is no Hive Metastore configured and no compression used.

See the documentation on the Schema setting to view an example of the schema.

 Parquet Writer using the Partition by

Example on "Partition by":

Assume the following input documents:

[
    {
        "month" : "MAR",
        "day" : "01",
        "msg" : "Hello, World",
        "num" : 1
    },
    {
        "month" : "FEB",
        "day" : "07",
        "msg" : "Hello, World",
        "num" : 3
    },
    {
        "month" : "MAR",
        "day" : "01",
        "msg" : "Hello, World",
        "num" : 2
    },
    {
        "month" : "FEB",
        "day" : "07",
        "msg" : "Hello, World",
        "num" : 4
    }
]

The settings of the Parquet Writer Snap are as follows:

The pipeline execution will generate two files:

hdfs://localhost:8080/tmp/FEB/07/sample.parquet

hdfs://localhost:8080/tmp/MAR/01/sample.parquet


The key-value pairs for "month" and "day" will not be included in the output files.

 Parquet Reader/Writer with S3 in Standard Mode

Reading and writing Parquet files from/to AWS S3 requires an S3 account.

  1. Create an S3 account or use an existing one.
    1. If it is a regular S3 account, name the account and supply the Access-key ID and Secret key.
    2. If the account is IAM role enabled Account:

      1. Select the IAM role checkbox.

      2. Leave the Access-key ID and Secret key blank.

      3. The IAM role properties are optional. You can leave them blank.

        To use IAM Role Properties, ensure to select the IAM Role check box.


  2. Within the Parquet Snap, use a valid S3 oath for the directory in the format of:
    s3://<bucket name>/<folder name>/.../<filename>



Inserting and Querying Custom Metadata from the Flight Metadata Table

The Pipeline in this zipped example, MetadataCatalog_Insert_Read_Example.zip, demonstrates how you can:

  • Use the Catalog Insert Snap to update metadata tables.
  • Use the Catalog Query Snap to read the updated metadata information.

In this example:

  1. We import a file containing the metadata.
  2. We create a parquet file using the data in the imported file
  3. We insert metadata that meets specific requirements into a partition in the target table.
  4. We read the newly-inserted metadata using the Catalog Query Snap.


 Understanding the Pipeline

The Pipeline is designed as follows:

The File Reader Snap read flight statistics and the JSON Parser Snap parses the data into a JSON file.

The Parquet Writer Snap creates a Parquet file with the data of the JSON file, in an S3 database.

The output of the Parquet Writer Snap includes the schema of the file. This is the metadata that must be included into the catalog.

The Catalog Insert Snap picks up the schema from the Parquet file and associates it with a specific partition in the target table. It also adds a custom property to the partition.

Once the Snap completes execution, the table is inserted into the metadata catalog and you can view the table in the SnapLogic Manager.

To view the table, navigate to the Project where you have created the Pipeline, click the Table tab, and then click the new table created after executing the Pipeline. This displays the table. Click Show schema to view the metadata.

The Schema view does not display the custom metadata that you inserted into the partition. Use the Catalog Query Snap to view all the updates made by the Catalog Insert Snap.

Download this ZIP file.

 How to use the Sample ZIP File

Working with the Sample ZIP File

This ZIP file contains two files:

  • Metadata_Catalog_Insert_Read.slp
  • AllDataTypes.json

To import this Pipeline:

  1. Download the ZIP file and extract its contents into a local directory.
  2. Import the Metadata_Catalog_Insert_Read.SLP Pipeline into a SnapLogic project.
  3. Open the Pipeline and click the File Reader Snap.
  4. In the File Reader Settings popup, use the  button to import and read the AllDataTypes.json file.
  5. Your Pipeline and test data are now ready. Review the other steps listed out in this example before validating or executing this Pipeline.

  File Modified

File Example_Parquet_Writer.slp

Aug 30, 2022 by Kalpana Malladi

Related Links

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
November 2022main18944 Stable

The AWS S3 and S3 Dynamic accounts now support a maximum session duration of an IAM role defined in AWS.

August 2022main17386 StableExtended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
4.29 Patch429patches16630 Latest
  • Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
  • Fixed an issue in the following Snaps that use AWS S3 dynamic account, where the Snaps displayed the security credentials like Access Key, Secret Key, and Security Token in the logs. Now, the security credentials in the logs are blurred for the Snaps that use AWS S3 dynamic account.
4.29main15993 Stable

Enhanced the AWS S3 Account for Hadoop account to include the S3 Region field that allows cross-region or proxied cross-region access to S3 buckets in the Parquet Reader and Parquet Writer Snaps.

4.28 Patch428patches15216 LatestAdded the AWS S3 Dynamic account for Parquet Reader and Parquet Writer Snaps.
4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches13769 Latest

Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM.

4.27 Patch427patches12999 LatestEnhanced the Parquet Reader Snap with int96 As Timestamp checkbox, which when selected enables the Date Time Format field. You can use this field to specify a date-time format of your choice for int96 data-type fields. The int96 As Timestamp checkbox is available only when you deselect Use old data format checkbox.

4.27

main12833

 

Stable

Enhanced the Parquet Writer and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration. 

4.26426patches12288 Latest

Fixed a memory leak issue when using HDFS protocol in Hadoop Snaps.

4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches9975 Latest

Fixed the dependency issue in Hadoop Parquet Reader Snap while reading from AWS S3. The issue is caused due to conflicting definitions for some of the AWS classes (dependencies) in the classpath.

4.25main9554
 
Stable
  • Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
    • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
    • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
4.24 Patch424patches9262 Latest

Enhanced the AWS S3 Account for Hadoop to support role-based access when you select IAM role checkbox.

4.24 Patch424patches8876

 


Latest

Fixes the missing library error in Hadoop Snap Pack when running Hadoop Pipelines in JDK11 runtime.

4.24main8556
StableUpgraded with the latest SnapLogic Platform release.
4.23 Patch423patches7440 Latest

Fixes the issue in HDFS Reader Snap by supporting to read and write files larger than 2GB using ABFS(S) protocol.

4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21 Patchhadoop8853 Latest

Updates the Parquet Writer and Parquet Reader Snaps to support the yyyy-MM-dd format for the DATE logical type.

4.21snapsmrc542

 

StableUpgraded with the latest SnapLogic Platform release.
4.20 Patchhadoop8776 Latest

Updates the Hadoop Snap Pack to use the latest version of org.xerial.snappy:snappy-java for compression type Snappy, in order to resolve the java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I error.

4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19 Patchhadoop8270 Latest

Fixes an issue with the Hadoop Parquet Writer Snap wherein the Snap throws an exception when the input document includes one or all of the following:

  • Empty lists.
  • Lists with all null values.
  • Maps with all null values.
4.19snaprsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18 Patchhadoop8033 Latest

Fixed an issue with the Parquet Writer Snap wherein the Snap throws an error when working with WASB protocol.

4.18snapsmrc523
 
Stable
4.17ALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508
 
Stable

Added a new property, Output for each file written, to handle multiple binary input data in the HDFS Writer Snap.

4.15snapsmrc500
 
Stable
  • Added two new Snaps: HDFS ZipFile Reader and HDFS ZipFile Writer.
  • Added support for the Data Catalog Snaps in Parquet Reader and Parquet Writer Snaps.
4.14 Patchhadoop5888 Latest

Fixed an issue wherein the Hadoop snaps were throwing an exception when a Kerberized account is provided, but the snap is run in a non-kerberized environment.

4.14snapsmrc490
 
Stable
  • Added the Hadoop Directory Browser Snap, which browses a given directory path in the Hadoop file system using the HDFS protocol and generates a list of all the files in the directory. It also lists subdirectories and their contents.
  • Added support for S3 file protocol in the ORC Reader, and ORC Writer Snaps.
  • Added support for reading nested schema in the Parquet Reader Snap.
4.13 Patchhadoop5318 Latest
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein Hadoop configuration information does not parse from the client's configuration files.
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein User Impersonation does not work on Hadooplex.
4.13

snapsmrc486

 
Stable
  • KMS encryption support added to AWS S3 account in the Hadoop Snap Pack.
  • Enhanced the Parquet Reader, Parquet Writer, HDFS Reader, and HDFS Writer Snaps to support WASB and ADLS file protocols.
  • Added the AWS S3 account support to the Parquet Reader and Writer Snaps. 
  • Added second input view to the Parquet Reader Snap that when enabled, accepts table schema.
  • Supported with AWS S3, Azure Data Lake, and Azure Storage Accounts.
4.12 Patchhadoop5132 Latest

Fixed an issue with the HDFS Reader Snap wherein the pipeline becomes stale while writing to the output view.

4.12

snapsmrc480

 
StableUpgraded with the latest SnapLogic Platform release.
4.11 Patchhadoop4275
Latest

Addressed an issue with Parquet Reader Snap leaking file descriptors (connections to HDFS data nodes). The Open File descriptor values work stable now,

4.11snapsmrc465
 
Stable

Added Kerberos support to the standard mode Parquet Reader and Parquet Writer Snaps.

4.10 Patchhadoop4001 Latest

Supported HDFS Writer to write to the encryption zone.

4.10 Patchhadoop3887 Latest

Addressed the suggest issue for the HDFS Reader on Hadooplex.

4.10 Patchhadoop3851 Latest
  • ORC supports read/write from local file system.
  • Addressed an issue to bind the Hive Metadata to Parquet Writer Schema at Runtime.
4.10 Patchhadoop3838 Latest

Made HDFS Snaps work with Zone encrypted HDFS.

4.10

snapsmrc414

 
Stable
  • Updated the Parquet Writer Snap with Partition by property to support the data written into HDFS based on the partition definition in the schema in Standard mode.
  • Support for S3 accounts with IAM Roles added to Parquet Reader and Parquet Writer
  • HDFS Reader/Writer with Kerberos support on Groundplex (including user impersonation).
4.9 Patchhadoop3339 Latest

Addressed the following issues:

  • ORC Reader passing, but ORC Writer failing when run on a Cloudplex.
  • ORC Reader Snap is not routing error to error view.
  • Intermittent failures with the ORC Writer
4.9.0 Patchhadoop3020 Latest

Added missing dependency org.iq80.snappy:snappy to Hadoop Snap Pack.

4.9snapsmrc405
 
StableUpgraded with the latest SnapLogic Platform release.
4.8

snapsmrc398

 
Stable

Snap-aware error handling policy enabled for Spark mode in Sequence Formatter and Sequence Parser. This ensures the error handling specified on the Snap is used.

4.7.0 Patchhadoop2343 Latest

Spark Validation: Resolved an issue with validation failing when setting the output file permissions.

4.7

snapsmrc382

 
Stable
  • Updated the HDFS Writer and HDFS Reader Snaps with Azure Data Lake account for standard mode pipelines.
  • HDFS Writer: Spark mode support added to write to a specified directory in an Azure Storage Layer using the wasb file system protocol.
  • HDFS Reader: Spark mode support added to read a single file or an HDFS directory from an Azure Storage Layer.
4.6snapsmrc362
 
Stable
  • The following Snaps now support error view in Spark mode: HDFS Reader, Sequence Parser.
  • Resolved an issue in HDFS Writer Snap that sends the same data in output & error view.
4.5

snapsmrc344

 
Stable
  • HDFS Reader and HDFS Writer Snaps updated to support IAM Roles for Amazon EC2.
  • Support for Spark mode added to Parquet Reader, Parquet Writer
  • The HBase Snaps are no longer available as of this release.
4.4.1
 Stable
  • Resolved an issue with Sequence Formatter not working in Spark mode.
  • Resolved an issue with HDFSReader not using the filter set when configuring SparkExec paths.
4.4
 Stable
  • NEW! Parquet Reader and Writer Snaps
  • NEW! ORC Reader and Writer Snaps
  • Spark support added to the HDFS Reader, HDFS Writer, Sequence Formatter, and Sequence Parser Snaps.
  • Behavior change: HDFS Writer in SnapReduce mode now requires the File property to be blank.
4.3.2
 Stable
  • Implemented wasbs:// protocol support in Hadoop Snap Pack.
  • Resolved an issue with HDFS Reader unable to read all files under a folder (including all files under its subfolders) using the ** filter.