Skip to end of banner
Go to start of banner

HDFS Writer

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 56 Next »

In this article

Overview

This Snap reads a binary data stream from its input view and writes a file in HDFS (Hadoop File System). It also helps pick a file by suggesting a list of directories and files. For the HDFS protocol, use a SnapLogic on-premises Groundplex and ensure its instance is within the Hadoop cluster and SSH authentication has already been established. The Snap also supports writing to a Kerberized cluster through the HDFS protocol. This Snap supports HDFS, ABFS (Azure Data Lake Storage Gen 2 ), and WASB (Azure storage) protocols. HDFS 2.4.0 is supported for the HDFS protocol. It also supports reading from HDFS Encryption.

hdfs-writer-overview.png

Snap Type

The HDFS Writer Snap is a Write-type Snap.

Prerequisites

None.

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

  • Append (File action) is supported for ADL protocol only.

  • File names with the following special characters are not supported in the HDFS Writer Snap: '+', '?', '/', ':'.

Known Issues

The upgrade of Azure Storage library from v3.0.0 to v8.3.0 has caused the following issue when using the WASB protocol:
When you use invalid credentials for the WASB protocol in Hadoop Snaps (HDFS Reader, HDFS Writer, ORC Reader, Parquet Reader, Parquet Writer), the pipeline does not fail immediately, instead it takes 13-14 minutes to display the following error:

reason=The request failed with error code null and HTTP code 0. , status_code=error

SnapLogic® is actively working with Microsoft®Support to resolve the issue.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input

Binary

  • Min: 1

  • Max: 1

  • CSV Formatter

  • JSON Formatter

  • XML Formatter

Binary input data.

Output

Document

  • Min: 0

  • Max: 1

  • Maper

  • File Reader

The following is an example of the output document map data:

{

"filename": "hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv", 3 "fileAction": "overwritten" }

The value of the "fileAction" field can be "overwritten" or "created" or "ignored". The value "ignored" indicate that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE".

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the field set.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the field set.

  • Upload icon ((blue star) ): Indicates that you can upload files.

Field

Field Types

Description

Label*


Default Value: HDFS Writer
Example: HDFS Writer

String

Specify a unique name for the Snap.

Directory


Default Value: hdfs://<hostname>:<port>/
Example:

  • hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/

  • wasb:///snaplogic/testDir/

  • wasbs:///snaplogic/testDir/

  • $dirname

  • adl://snapqa/ 

  • abfs(s):///filesystem2/dir1

  • abfs(s)://filesystem2@snaplogicaccount.dfs.core.windows.net/dir1

String/Expression/Suggestion

Specify the URL for HDFS directory. It should start with hdfs file protocol in the following format:

  • hdfs://<hostname>:<port>/<path to directory>/

  • wasb:///<container name>/<path to directory>/

  • wasbs:///<container name>/<path to directory>/

  • adl://<container name>/<path to directory>/ 

  • abfs(s):///filesystem/<path>/

  • abfs(s)://filesystem@accountname.endpoint/<path>

The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click the Suggestion icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.

SnapLogic automatically appends azuredatalakestore.net to the store name you specify when using Azure Data Lake; therefore, you do not have to add azuredatalakestore.net to the URI while specifying the directory.

File filter


Default Value: *


String/Expression

Specify the Glob filter pattern.

Use glob patterns to display a list of directories or files when you click the Suggest icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.

 Glob Pattern Interpretation Rules

The following rules are used to interpret glob patterns:

☑️ The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period.

☑️ The ** characters match zero or more characters across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory.

☑️ The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension.

☑️ The \ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash, and \{ matches a left brace, for example.

☑️ The ! character is used to exclude matching files from the output. 

☑️ The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'.

☑️ Within a bracket expression, the '*', '?', and '\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating.

☑️ The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'.

☑️ Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name ".login".

☑️ All other characters match themselves.

Examples:

'*.csv' matches all files with a csv extension in the current directory only.

'**.csv' matches all files with a csv extension in the current directory and in all its subdirectories.

*[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP.

File

Default Value: N/A
Example:

  • sample.csv

  • tmp/another.csv

  • $filename

String/Expression/Suggestion

Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". The File property can be a JavaScript expression which will be evaluated with values from the input view document. When you click the Suggest icon, it will display a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.

Flush interval (kB)

Default Value: -1
Example: 0

Interval

Specify the flush interval in kilobytes to flush a specified size of data during the file upload. This Snap can flush the output stream each time a given size of data is written to the target file server.

If the Flush interval is 0, the Snap flushes maximum frequency after each byte block is written. The larger the flush interval, the less frequent are the flushes. This field is useful if the file upload experiences an intermittent failure. However, more frequent flushes result in slower file upload. The default value of -1 indicates no flush during the upload.

Number Of Retries

Default Value: 0
Example: 1

Integer/Expression

Specify the maximum number of attempts to be made to receive a response. 

  • The request is terminated if the attempts do not result in a response.

  • Retry operation occurs only when the Snap loses the connection with the server.

Retry Interval (seconds)

Default Value1
Example: 5

Integer/Expression

Specify the time interval between two successive retry requests. A retry happens only when the previous attempt resulted in an exception.

File action*


Default ValueOverwrite
Example: Append 


Dropdown list

Select an action to perform if the specified file already exists:

  • Overwrite - The Snap attempts to write the file without checking for the file's existence for a better performance, and the "fileAction" field will be "overwritten" in the output view data.

  • Append - The Snap appends records in the incoming documents to the existing file.

  • Ignore - If the file already exists, the Snap does not throw an exception and does not overwrite the file, but writes an output document indicating that it has been 'ignored'.

  • Error - The error displays in the Pipeline Run Log if the file already exists.

The Append operation is supported for FILE, SFTP, FTP, FTPS and ADL protocols only. For any other protocols that are not supported by Append, we recommend that you use the File Operations, File Writer, and File Delete Snaps and follow this procedure. 

  1. Copy the blob file to your local drive from the endpoint.

  2. Append additional data to the local file.

  3. Delete the original file on the endpoint.

  4. Copy the modified temporary file back from the source to the target.

Note: This approach might involve disk overhead, therefore ensure that you have enough disk space in your system.

File permissions for various users

Use this field set to select the user and the desired file permissions.

Limitations with File Permissions for Various Users

  • With "File permissions for various users": File names with the following special characters are not supported in the HDFS Writer Snap: '+', '?', '/', ':'.

  • Without "File permissions for various users": File names with the following special characters are not supported in the HDFS Writer Snap: ':', '/'.

User type

Default Value: N/A
Example:  owner, group, others

String/Expression/Suggestion

It should be 'owner' or 'group' or 'others'. Each row can have only one user type and each user type should appear only once. Please select one from the suggested list.

File permissions

Default Value: N/A
Example:  read, write, execute, read+write, read+write+execute

String/Expression/Suggestion

It can be any combination of {read, write, execute} separated by '+' character. Please select one from the suggested list.

User Impersonation


Default Value: Deselected

Checkbox

Select this check box to enable user impersonation.

Hadoop allows you to configure proxy users to access HDFS on behalf of other users; this is called impersonation. When user impersonation is enabled on the Hadoop cluster, any jobs submitted using a proxy are executed with the impersonated user's existing privilege levels rather than those of the superuser associated with the cluster. For more information on user impersonation in this Snap, see the section on User Impersonation below.

Output for each file written


Default Value: Deselected

Checkbox

Enables you to produce a different output document for each file that is written. If the Snap receives multiple binary input data and the File expression property is dynamically evaluated to a filename by using the Content-Location field from the input metadata, each binary data can be written to a different target file.

By default, the Snap produces only one output document with a filename that corresponds to the last file that was written.

Write empty file

 

Default Value: Deselected

Checkbox

Select this checkbox to write an empty file to all the supported protocols when the binary input document has no data.

Snap Execution


Default Value: Validate & Execute
Example: Execute only

Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Troubleshooting

Writing to S3 files with HDFS version CDH 5.8 or later

When running HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:

  1. Go to HDFS configuration.
  2. In Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, add an entry with the following details:
    • Name: fs.s3a.threads.max
    • Value: 15
  3. Click Save.
  4. Restart all the nodes.
  5. Under Restart Stale Services, select Re-deploy client configuration.
  6. Click Restart Now.


Snap Pack History

Hadoop Snap Pack

 Click to view/expand
Release Snap Pack VersionDateType  Updates

August 2024

main27765

 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

May 2024437patches27226 -

The upgrade of the Azure Storage library from v3.0.0 to v8.3.0 has impacted the Hadoop Snap Pack causing the following issue when using the WASB protocol.

Known Issue

When you use invalid credentials for the WASB protocol in Hadoop Snaps (HDFS Reader, HDFS Writer, ORC Reader, Parquet Reader, Parquet Writer), the pipeline does not fail immediately, instead it takes 13-14 minutes to display the following error:

reason=The request failed with error code null and HTTP code 0. , status_code=error

SnapLogic® is actively working with Microsoft®Support to resolve the issue.

Learn more about the Azure Storage Library Upgrade.

May 2024437patches27471 Latest

Fixed a resource leak issue with the following Hadoop Snaps, which involved too many stale instances of ProxyConnectionManager and significantly impacted memory utilization.

May 2024437patches26370 Latest

Enhanced the HDFS Writer Snap with the Write empty file checkbox to enable you to write an empty or a 0-byte file to all the supported protocols that are recognized and compatible with the target system or destination.

May 2024main26341 Stable

The Azure Data Lake Account has been removed from the Hadoop Snap Pack because Microsoft retired the Azure Data Lake Storage Gen1 protocol on February 29, 2024. We recommend replacing your existing Azure Data Lake Accounts (in Binary or Hadoop Snap Packs) with other Azure Accounts.

February 2024

436patches25902

 Latest

Fixed a memory management issue in the HDFS Writer, HDFS ZipFile Writer, ORC Writer, and Parquet Writer Snaps, which previously caused out-of-memory errors when multiple Snaps were used in the pipeline. The Snap now conducts a pre-allocation memory check, dynamically adjusting the write buffer size based on available memory resources when writing to ADLS.

February 2024435patches25410Latest

Enhanced the AWS S3 Account for Hadoop with an External ID that enables you to access Hadoop resources securely.

February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023435patches23904 Latest
  • Fixed an issue with the Parquet Writer Snap that displayed an error Failed to write parquet data when the decimal value passed from the second input view exceeded the specified scale.

  • Fixed an issue with the Parquet Writer Snap that previously failed to handle the conversion of BigInt/int64 (larger numbers) after the 4.35 GA now converts them accurately.

November 2023435patches23780 Latest

Fixed an issue related to error routing to the output view. Also fixed a connection timeout issue.

November 2023main23721StableUpdated and certified against the current SnapLogic Platform release.
August 2023434patches23173 Latest

Enhanced the Parquet Writer Snap with a Decimal Rounding Mode dropdown list to enable the rounding method for decimal values when the number exceeds the required decimal places.

August 2023434patches22662 Latest
  • Enhanced the Parquet Writer Snap with the support for LocalDate and DateTime. The Snap now shows the schema suggestions for LocalDate and DateTime correctly.

  • Enhanced the Parquet Reader Snap with the Use datetime types checkbox that supports LocalDate and DateTime datatypes.

Behavior change:

When you select the Use datetime types checkbox in the Parquet Reader Snap, the Snap displays the LocalDate and DateTime in the output for INT32 (DATE) and INT64 (TIMESTAMP_MILLIS) columns. When you deselect this checkbox, the columns retain the previous datatypes and display string and integer values in the output.

August 2023

main22460

 

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023433patches22180 Latest

Introduced the HDFS Delete, which deletes the specified file, group of files, or directory from the supplied path and protocol in the Hadoop Distributed File System (HDFS).

May 2023433patches21494 LatestThe Hadoop Directory Browser Snap now returns all the output documents as expected after implementing pagination for the ABFS protocol.
May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023

432patches20820

 Latest

Fixed an authorization issue that occurs with the Parquet Writer Snap when it receives empty document input.

February 2023432patches20209 Latest

The Apache Commons Compress library has been upgraded to version 1.22.

February 2023432patches20139 Latest

The Kerberos Account that is available for a subset of snaps in the Hadoop Snap pack now supports a configuration that enables you to read from and write to the Hadoop Distributed File System (HDFS) managed by multiple Hadoop clusters. You can specify the location of the Hadoop configuration files in the Hadoop config directory field. The value in this field overrides the value that is set on the Snaplex system property used for configuring a single cluster. 

February 2023

main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 Stable

The AWS S3 and S3 Dynamic accounts now support a maximum session duration of an IAM role defined in AWS.

August 2022main17386 StableExtended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
4.29 Patch429patches16630 Latest
  • Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
  • Fixed an issue in the following Snaps that use AWS S3 dynamic account, where the Snaps displayed the security credentials like Access Key, Secret Key, and Security Token in the logs. Now, the security credentials in the logs are blurred for the Snaps that use AWS S3 dynamic account.
4.29main15993 Stable

Enhanced the AWS S3 Account for Hadoop account to include the S3 Region field that allows cross-region or proxied cross-region access to S3 buckets in the Parquet Reader and Parquet Writer Snaps.

4.28 Patch428patches15216 LatestAdded the AWS S3 Dynamic account for Parquet Reader and Parquet Writer Snaps.
4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches13769 Latest

Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM.

4.27 Patch427patches12999 LatestEnhanced the Parquet Reader Snap with int96 As Timestamp checkbox, which when selected enables the Date Time Format field. You can use this field to specify a date-time format of your choice for int96 data-type fields. The int96 As Timestamp checkbox is available only when you deselect Use old data format checkbox.

4.27

main12833

 

Stable

Enhanced the Parquet Writer and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration. 

4.26426patches12288 Latest

Fixed a memory leak issue when using HDFS protocol in Hadoop Snaps.

4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches9975 Latest

Fixed the dependency issue in Hadoop Parquet Reader Snap while reading from AWS S3. The issue is caused due to conflicting definitions for some of the AWS classes (dependencies) in the classpath.

4.25main9554
 
Stable
  • Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
    • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
    • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
4.24 Patch424patches9262 Latest

Enhanced the AWS S3 Account for Hadoop to support role-based access when you select IAM role checkbox.

4.24 Patch424patches8876

 


Latest

Fixes the missing library error in Hadoop Snap Pack when running Hadoop Pipelines in JDK11 runtime.

4.24main8556
StableUpgraded with the latest SnapLogic Platform release.
4.23 Patch423patches7440 Latest

Fixes the issue in HDFS Reader Snap by supporting to read and write files larger than 2GB using ABFS(S) protocol.

4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21 Patchhadoop8853 Latest

Updates the Parquet Writer and Parquet Reader Snaps to support the yyyy-MM-dd format for the DATE logical type.

4.21snapsmrc542

 

StableUpgraded with the latest SnapLogic Platform release.
4.20 Patchhadoop8776 Latest

Updates the Hadoop Snap Pack to use the latest version of org.xerial.snappy:snappy-java for compression type Snappy, in order to resolve the java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I error.

4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19 Patchhadoop8270 Latest

Fixes an issue with the Hadoop Parquet Writer Snap wherein the Snap throws an exception when the input document includes one or all of the following:

  • Empty lists.
  • Lists with all null values.
  • Maps with all null values.
4.19snaprsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18 Patchhadoop8033 Latest

Fixed an issue with the Parquet Writer Snap wherein the Snap throws an error when working with WASB protocol.

4.18snapsmrc523
 
Stable
4.17ALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508
 
Stable

Added a new property, Output for each file written, to handle multiple binary input data in the HDFS Writer Snap.

4.15snapsmrc500
 
Stable
  • Added two new Snaps: HDFS ZipFile Reader and HDFS ZipFile Writer.
  • Added support for the Data Catalog Snaps in Parquet Reader and Parquet Writer Snaps.
4.14 Patchhadoop5888 Latest

Fixed an issue wherein the Hadoop snaps were throwing an exception when a Kerberized account is provided, but the snap is run in a non-kerberized environment.

4.14snapsmrc490
 
Stable
  • Added the Hadoop Directory Browser Snap, which browses a given directory path in the Hadoop file system using the HDFS protocol and generates a list of all the files in the directory. It also lists subdirectories and their contents.
  • Added support for S3 file protocol in the ORC Reader, and ORC Writer Snaps.
  • Added support for reading nested schema in the Parquet Reader Snap.
4.13 Patchhadoop5318 Latest
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein Hadoop configuration information does not parse from the client's configuration files.
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein User Impersonation does not work on Hadooplex.
4.13

snapsmrc486

 
Stable
  • KMS encryption support added to AWS S3 account in the Hadoop Snap Pack.
  • Enhanced the Parquet Reader, Parquet Writer, HDFS Reader, and HDFS Writer Snaps to support WASB and ADLS file protocols.
  • Added the AWS S3 account support to the Parquet Reader and Writer Snaps. 
  • Added second input view to the Parquet Reader Snap that when enabled, accepts table schema.
  • Supported with AWS S3, Azure Data Lake, and Azure Storage Accounts.
4.12 Patchhadoop5132 Latest

Fixed an issue with the HDFS Reader Snap wherein the pipeline becomes stale while writing to the output view.

4.12

snapsmrc480

 
StableUpgraded with the latest SnapLogic Platform release.
4.11 Patchhadoop4275
Latest

Addressed an issue with Parquet Reader Snap leaking file descriptors (connections to HDFS data nodes). The Open File descriptor values work stable now,

4.11snapsmrc465
 
Stable

Added Kerberos support to the standard mode Parquet Reader and Parquet Writer Snaps.

4.10 Patchhadoop4001 Latest

Supported HDFS Writer to write to the encryption zone.

4.10 Patchhadoop3887 Latest

Addressed the suggest issue for the HDFS Reader on Hadooplex.

4.10 Patchhadoop3851 Latest
  • ORC supports read/write from local file system.
  • Addressed an issue to bind the Hive Metadata to Parquet Writer Schema at Runtime.
4.10 Patchhadoop3838 Latest

Made HDFS Snaps work with Zone encrypted HDFS.

4.10

snapsmrc414

 
Stable
  • Updated the Parquet Writer Snap with Partition by property to support the data written into HDFS based on the partition definition in the schema in Standard mode.
  • Support for S3 accounts with IAM Roles added to Parquet Reader and Parquet Writer
  • HDFS Reader/Writer with Kerberos support on Groundplex (including user impersonation).
4.9 Patchhadoop3339 Latest

Addressed the following issues:

  • ORC Reader passing, but ORC Writer failing when run on a Cloudplex.
  • ORC Reader Snap is not routing error to error view.
  • Intermittent failures with the ORC Writer
4.9.0 Patchhadoop3020 Latest

Added missing dependency org.iq80.snappy:snappy to Hadoop Snap Pack.

4.9snapsmrc405
 
StableUpgraded with the latest SnapLogic Platform release.
4.8

snapsmrc398

 
Stable

Snap-aware error handling policy enabled for Spark mode in Sequence Formatter and Sequence Parser. This ensures the error handling specified on the Snap is used.

4.7.0 Patchhadoop2343 Latest

Spark Validation: Resolved an issue with validation failing when setting the output file permissions.

4.7

snapsmrc382

 
Stable
  • Updated the HDFS Writer and HDFS Reader Snaps with Azure Data Lake account for standard mode pipelines.
  • HDFS Writer: Spark mode support added to write to a specified directory in an Azure Storage Layer using the wasb file system protocol.
  • HDFS Reader: Spark mode support added to read a single file or an HDFS directory from an Azure Storage Layer.
4.6snapsmrc362
 
Stable
  • The following Snaps now support error view in Spark mode: HDFS Reader, Sequence Parser.
  • Resolved an issue in HDFS Writer Snap that sends the same data in output & error view.
4.5

snapsmrc344

 
Stable
  • HDFS Reader and HDFS Writer Snaps updated to support IAM Roles for Amazon EC2.
  • Support for Spark mode added to Parquet Reader, Parquet Writer
  • The HBase Snaps are no longer available as of this release.
4.4.1
 Stable
  • Resolved an issue with Sequence Formatter not working in Spark mode.
  • Resolved an issue with HDFSReader not using the filter set when configuring SparkExec paths.
4.4
 Stable
  • NEW! Parquet Reader and Writer Snaps
  • NEW! ORC Reader and Writer Snaps
  • Spark support added to the HDFS Reader, HDFS Writer, Sequence Formatter, and Sequence Parser Snaps.
  • Behavior change: HDFS Writer in SnapReduce mode now requires the File property to be blank.
4.3.2
 Stable
  • Implemented wasbs:// protocol support in Hadoop Snap Pack.
  • Resolved an issue with HDFS Reader unable to read all files under a folder (including all files under its subfolders) using the ** filter.
  • No labels