RC File Formatter

August 2024

main27765

21 Aug 2024 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

Fixed a resource leak issue with the following Hadoop Snaps, which involved too many stale instances of ProxyConnectionManager and significantly impacted memory utilization.

• Parquet Reader

• Parquet Writer

• ORC Reader

• ORC Writer

• HDFS ZipFile Reader

• HDFS ZipFile Writer

Enhanced the HDFS Writer Snap with the Write empty file checkbox to enable you to write an empty or a 0-byte file to all the supported protocols that are recognized and compatible with the target system or destination.

The Azure Data Lake Account has been removed from the Hadoop Snap Pack because Microsoft retired the Azure Data Lake Storage Gen1 protocol on February 29, 2024. We recommend replacing your existing Azure Data Lake Accounts (in Binary or Hadoop Snap Packs) with other Azure Accounts.

436patches25902

Fixed a memory management issue in the HDFS Writer, HDFS ZipFile Writer, ORC Writer, and Parquet Writer Snaps, which previously caused out-of-memory errors when multiple Snaps were used in the pipeline. The Snap now conducts a pre-allocation memory check, dynamically adjusting the write buffer size based on available memory resources when writing to ADLS.

Enhanced the AWS S3 Account for Hadoop with an External ID that enables you to access Hadoop resources securely.

• Fixed an issue with the Parquet Writer Snap that displayed an error Failed to write parquet data when the decimal value passed from the second input view exceeded the specified scale.

• Fixed an issue with the Parquet Writer Snap that previously failed to handle the conversion of BigInt/int64 (larger numbers) after the 4.35 GA now converts them accurately.

Fixed an issue related to error routing to the output view. Also fixed a connection timeout issue.

Enhanced the Parquet Writer Snap with a Decimal Rounding Mode dropdown list to enable the rounding method for decimal values when the number exceeds the required decimal places.

• Enhanced the Parquet Writer Snap with the support for LocalDate and DateTime. The Snap now shows the schema suggestions for LocalDate and DateTime correctly.

• Enhanced the Parquet Reader Snap with the Use datetime types checkbox that supports LocalDate and DateTime datatypes.

Behavior change:

When you select the Use datetime types checkbox in the Parquet Reader Snap, the Snap displays the LocalDate and DateTime in the output for INT32 (DATE) and INT64 (TIMESTAMP_MILLIS) columns. When you deselect this checkbox, the columns retain the previous datatypes and display string and integer values in the output.

August 2023

main22460

Stable

Updated and certified against the current SnapLogic Platform release.

Introduced the HDFS Delete, which deletes the specified file, group of files, or directory from the supplied path and protocol in the Hadoop Distributed File System (HDFS).

432patches20820

Fixed an authorization issue that occurs with the Parquet Writer Snap when it receives empty document input.

The Apache Commons Compress library has been upgraded to version 1.22.

The Kerberos Account that is available for a subset of snaps in the Hadoop Snap pack now supports a configuration that enables you to read from and write to the Hadoop Distributed File System (HDFS) managed by multiple Hadoop clusters. You can specify the location of the Hadoop configuration files in the Hadoop config directory field. The value in this field overrides the value that is set on the Snaplex system property used for configuring a single cluster. 

The AWS S3 and S3 Dynamic accounts now support a maximum session duration of an IAM role defined in AWS.

• Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
• Fixed an issue in the following Snaps that use AWS S3 dynamic account, where the Snaps displayed the security credentials like Access Key, Secret Key, and Security Token in the logs. Now, the security credentials in the logs are blurred for the Snaps that use AWS S3 dynamic account.

  • ORC Reader

  • ORC Writer

  • Parquet Reader

  • Parquet Writer

Enhanced the AWS S3 Account for Hadoop account to include the S3 Region field that allows cross-region or proxied cross-region access to S3 buckets in the Parquet Reader and Parquet Writer Snaps.

Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM.

4.27

main12833

13 Nov 2021 

Stable

Enhanced the Parquet Writer and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration. 

Fixed a memory leak issue when using HDFS protocol in Hadoop Snaps.

Fixed the dependency issue in Hadoop Parquet Reader Snap while reading from AWS S3. The issue is caused due to conflicting definitions for some of the AWS classes (dependencies) in the classpath.

• Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
  • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
  • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.

Enhanced the AWS S3 Account for Hadoop to support role-based access when you select IAM role checkbox.

11 Mar 2021 

Fixes the missing library error in Hadoop Snap Pack when running Hadoop Pipelines in JDK11 runtime.

Fixes the issue in HDFS Reader Snap by supporting to read and write files larger than 2GB using ABFS(S) protocol.

Updates the Parquet Writer and Parquet Reader Snaps to support the yyyy-MM-dd format for the DATE logical type.

Updates the Hadoop Snap Pack to use the latest version of org.xerial.snappy:snappy-java for compression type Snappy, in order to resolve the java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I error.

Fixes an issue with the Hadoop Parquet Writer Snap wherein the Snap throws an exception when the input document includes one or all of the following:

• Empty lists.
• Lists with all null values.
• Maps with all null values.

Fixed an issue with the Parquet Writer Snap wherein the Snap throws an error when working with WASB protocol.

• Added ADLS Gen2 support for ABFS (Azure Blob File System) and ABFSS protocols to the HDFS ZipFile Reader, HDFS ZipFile Writer, HDFS Reader, ORC Reader, Parquet Reader, and Hadoop Directory Browser Snaps.
• Added ADLS Gen2 support for ABFS (Azure Blob File System) and ABFSS protocols.

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

Added a new property, Output for each file written, to handle multiple binary input data in the HDFS Writer Snap.

• Added two new Snaps: HDFS ZipFile Reader and HDFS ZipFile Writer.
• Added support for the Data Catalog Snaps in Parquet Reader and Parquet Writer Snaps.

Fixed an issue wherein the Hadoop snaps were throwing an exception when a Kerberized account is provided, but the snap is run in a non-kerberized environment.

• Added the Hadoop Directory Browser Snap, which browses a given directory path in the Hadoop file system using the HDFS protocol and generates a list of all the files in the directory. It also lists subdirectories and their contents.
• Added support for S3 file protocol in the ORC Reader, and ORC Writer Snaps.
• Added support for reading nested schema in the Parquet Reader Snap.

• Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein Hadoop configuration information does not parse from the client's configuration files.
• Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein User Impersonation does not work on Hadooplex.

snapsmrc486

• KMS encryption support added to AWS S3 account in the Hadoop Snap Pack.
• Enhanced the Parquet Reader, Parquet Writer, HDFS Reader, and HDFS Writer Snaps to support WASB and ADLS file protocols.
• Added the AWS S3 account support to the Parquet Reader and Writer Snaps. 
• Added second input view to the Parquet Reader Snap that when enabled, accepts table schema.
• Supported with AWS S3, Azure Data Lake, and Azure Storage Accounts.

Fixed an issue with the HDFS Reader Snap wherein the pipeline becomes stale while writing to the output view.

snapsmrc480

Addressed an issue with Parquet Reader Snap leaking file descriptors (connections to HDFS data nodes). The Open File descriptor values work stable now,

Added Kerberos support to the standard mode Parquet Reader and Parquet Writer Snaps.

Supported HDFS Writer to write to the encryption zone.

Addressed the suggest issue for the HDFS Reader on Hadooplex.

• ORC supports read/write from local file system.
• Addressed an issue to bind the Hive Metadata to Parquet Writer Schema at Runtime.

Made HDFS Snaps work with Zone encrypted HDFS.

snapsmrc414

• Updated the Parquet Writer Snap with Partition by property to support the data written into HDFS based on the partition definition in the schema in Standard mode.
• Support for S3 accounts with IAM Roles added to Parquet Reader and Parquet Writer
• HDFS Reader/Writer with Kerberos support on Groundplex (including user impersonation).

Addressed the following issues:

• ORC Reader passing, but ORC Writer failing when run on a Cloudplex.
• ORC Reader Snap is not routing error to error view.
• Intermittent failures with the ORC Writer

Added missing dependency org.iq80.snappy:snappy to Hadoop Snap Pack.

snapsmrc398

Snap-aware error handling policy enabled for Spark mode in Sequence Formatter and Sequence Parser. This ensures the error handling specified on the Snap is used.

Spark Validation: Resolved an issue with validation failing when setting the output file permissions.

snapsmrc382

• Updated the HDFS Writer and HDFS Reader Snaps with Azure Data Lake account for standard mode pipelines.
• HDFS Writer: Spark mode support added to write to a specified directory in an Azure Storage Layer using the wasb file system protocol.
• HDFS Reader: Spark mode support added to read a single file or an HDFS directory from an Azure Storage Layer.

• The following Snaps now support error view in Spark mode: HDFS Reader, Sequence Parser.
• Resolved an issue in HDFS Writer Snap that sends the same data in output & error view.

snapsmrc344

• HDFS Reader and HDFS Writer Snaps updated to support IAM Roles for Amazon EC2.
• Support for Spark mode added to Parquet Reader, Parquet Writer
• The HBase Snaps are no longer available as of this release.

• Resolved an issue with Sequence Formatter not working in Spark mode.
• Resolved an issue with HDFSReader not using the filter set when configuring SparkExec paths.

• NEW! Parquet Reader and Writer Snaps
• NEW! ORC Reader and Writer Snaps
• Spark support added to the HDFS Reader, HDFS Writer, Sequence Formatter, and Sequence Parser Snaps.
• Behavior change: HDFS Writer in SnapReduce mode now requires the File property to be blank.

• Implemented wasbs:// protocol support in Hadoop Snap Pack.
• Resolved an issue with HDFS Reader unable to read all files under a folder (including all files under its subfolders) using the ** filter.