/
Configuring Hadoop Accounts

Configuring Hadoop Accounts

Owned by Kalpana Malladi
Last updated: Mar 04, 2024 by Kalpana Malladi

In this article

Overview

You must configure accounts in Hadoop Snaps to enable them to connect to Hadoop tables.

In this section

Snap Compatibility

Snaps in the Hadoop Snap Pack work with different accounts and protocols per the following table:

SnapAWS S3 Account for HadoopAWS S3 Dynamic AccountAzure Storage for HadoopAzure Data LakeAzure Data Lake Storage Gen 2 OAuth2KerberosProtocols Supported
Hadoop Directory Browser✖✖✖✖

✔

✔
  • ABFS
  • HDFS
HDFS Reader✖✖✔✔✔✔
  • WASB
  • WASBS
  • ABFS
  • HDFS
  • ADL
HDFS Writer✖✖✔✔✔✔WASB, WASBS, ABFS, HDFS, ADL
HDFS ZipFile Reader✖✖✖✖✔✔ABFS, HDFS
HDFS ZipFile Writer✖✖✖✖✔✔ABFS, HDFS
ORC Reader✔✔✖✖✔✖AWS S3, WASB, WASBS, ABFS, HDFS
ORC Writer✔✔✖✖✔✖AWS S3, ABFS, HDFS
Parquet Reader✔✔✔✔✔✔AWS S3, WASB, WASBS, ABFS, HDFS, ADL
Parquet Writer✔✔✔✔✔✔AWS S3, WASB, WASBS, ABFS, HDFS, ADL
RC File Formatter, RC File Parser, Sequence Formatter, Sequence Parser✖✖✖✖✖✖N/A

Configuring Hadoop Accounts

You can configure Hadoop accounts in SnapLogic using either the Designer or Manager.

Using SnapLogic Designer

Selecting an existing account

SnapLogic organizes and displays all accounts to which you have access, sorting them by account type and location. To select an existing account:

  1. Click the  icon to view the accounts to which you have access and select the account that you want to use. 

  2. Click .

Creating an account

  1. Click Add Account in the Account Reference dialog.

  2. Select the Location in which you want to create the account, select the account type, and click Continue. The Add Account dialog associated with the account type appears.



  3. Enter the required account details. For detailed guidance on how to provide information associated with each account type, use the following links:

    Enter additional information on this account in the Notes field of the Info tab. This will help you–and other users–understand the purpose of the account, especially if there are multiple accounts of the same type.

  4. Click Validate to verify the account, if the account type supports validation.

  5. Click Apply to complete configuring the binary account.

Using SnapLogic Manager

Use Manager to create accounts without associating them immediately with Pipelines.

Accounts in SnapLogic are associated with projects. You can use accounts created in other projects only if you have at least Read access to them.

  1. In the left pane, browse to the project in which you want to create the account and click Create > Account > Binary, followed by the appropriate account type.


  2. The Create Account dialog associated with the selected account type appears. 

  3. Repeat the steps numbered 3 through 5 in the previous section.

Avoid updating account credentials while Pipelines using that account are executing. This may lead to unexpected results, including locking your account.


 Click to view/expand
Release Snap Pack VersionDateType  Updates
November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.

August 2024

main27765

 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

May 2024437patches27226 -

The upgrade of the Azure Storage library from v3.0.0 to v8.3.0 has impacted the Hadoop Snap Pack causing the following issue when using the WASB protocol.

Known Issue

When you use invalid credentials for the WASB protocol in Hadoop Snaps (HDFS Reader, HDFS Writer, ORC Reader, Parquet Reader, Parquet Writer), the pipeline does not fail immediately, instead it takes 13-14 minutes to display the following error:

reason=The request failed with error code null and HTTP code 0. , status_code=error

SnapLogic® is actively working with Microsoft®Support to resolve the issue.

Learn more about the Azure Storage Library Upgrade.

May 2024437patches27471 Latest

Fixed a resource leak issue with the following Hadoop Snaps, which involved too many stale instances of ProxyConnectionManager and significantly impacted memory utilization.

May 2024437patches26370 Latest

Enhanced the HDFS Writer Snap with the Write empty file checkbox to enable you to write an empty or a 0-byte file to all the supported protocols that are recognized and compatible with the target system or destination.

May 2024main26341 Stable

The Azure Data Lake Account has been removed from the Hadoop Snap Pack because Microsoft retired the Azure Data Lake Storage Gen1 protocol on February 29, 2024. We recommend replacing your existing Azure Data Lake Accounts (in Binary or Hadoop Snap Packs) with other Azure Accounts.

February 2024

436patches25902

 Latest

Fixed a memory management issue in the HDFS Writer, HDFS ZipFile Writer, ORC Writer, and Parquet Writer Snaps, which previously caused out-of-memory errors when multiple Snaps were used in the pipeline. The Snap now conducts a pre-allocation memory check, dynamically adjusting the write buffer size based on available memory resources when writing to ADLS.

February 2024435patches25410Latest

Enhanced the AWS S3 Account for Hadoop with an External ID that enables you to access Hadoop resources securely.

February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023435patches23904 Latest

  • Fixed an issue with the Parquet Writer Snap that displayed an error Failed to write parquet data when the decimal value passed from the second input view exceeded the specified scale.

  • Fixed an issue with the Parquet Writer Snap that previously failed to handle the conversion of BigInt/int64 (larger numbers) after the 4.35 GA now converts them accurately.

November 2023435patches23780 Latest

Fixed an issue related to error routing to the output view. Also fixed a connection timeout issue.

November 2023main23721StableUpdated and certified against the current SnapLogic Platform release.
August 2023434patches23173 Latest

Enhanced the Parquet Writer Snap with a Decimal Rounding Mode dropdown list to enable the rounding method for decimal values when the number exceeds the required decimal places.

August 2023434patches22662 Latest

  • Enhanced the Parquet Writer Snap with the support for LocalDate and DateTime. The Snap now shows the schema suggestions for LocalDate and DateTime correctly.

  • Enhanced the Parquet Reader Snap with the Use datetime types checkbox that supports LocalDate and DateTime datatypes.

Behavior change:

When you select the Use datetime types checkbox in the Parquet Reader Snap, the Snap displays the LocalDate and DateTime in the output for INT32 (DATE) and INT64 (TIMESTAMP_MILLIS) columns. When you deselect this checkbox, the columns retain the previous datatypes and display string and integer values in the output.

August 2023

main22460

 

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023433patches22180 Latest

Introduced the HDFS Delete, which deletes the specified file, group of files, or directory from the supplied path and protocol in the Hadoop Distributed File System (HDFS).

May 2023433patches21494 LatestThe Hadoop Directory Browser Snap now returns all the output documents as expected after implementing pagination for the ABFS protocol.
May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023

432patches20820

 Latest

Fixed an authorization issue that occurs with the Parquet Writer Snap when it receives empty document input.

February 2023432patches20209 Latest

The Apache Commons Compress library has been upgraded to version 1.22.

February 2023432patches20139 Latest

The Kerberos Account that is available for a subset of snaps in the Hadoop Snap pack now supports a configuration that enables you to read from and write to the Hadoop Distributed File System (HDFS) managed by multiple Hadoop clusters. You can specify the location of the Hadoop configuration files in the Hadoop config directory field. The value in this field overrides the value that is set on the Snaplex system property used for configuring a single cluster. 

February 2023

main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 Stable

The AWS S3 and S3 Dynamic accounts now support a maximum session duration of an IAM role defined in AWS.

August 2022main17386 StableExtended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
4.29 Patch429patches16630 Latest
  • Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
  • Fixed an issue in the following Snaps that use AWS S3 dynamic account, where the Snaps displayed the security credentials like Access Key, Secret Key, and Security Token in the logs. Now, the security credentials in the logs are blurred for the Snaps that use AWS S3 dynamic account.
4.29main15993 Stable

Enhanced the AWS S3 Account for Hadoop account to in