HDFS ZipFile Reader

On this Page


Use the HDFS ZipFile Read Snap to extract and read archive files in HDFS directories and produce a stream of unzipped documents in the output.

For the HDFS protocol, use a SnapLogic on-premises Groundplex. Also, ensure that the instance is within the Hadoop cluster and that SSH authentication is established.

This Snap supports the HDFS 2.4.0 protocol.

Expected Input and Output

  • Expected Input: Documents containing information that identifies the directory and ZIP files that must be read.
  • Expected Output: A binary stream containing unzipped documents from the specified ZIP files.
  • Expected Upstream Snaps: Required. Any Snap that offers a list of ZIP files in its output view. Examples: HDFS ZipFile Writer, ZipFile Read.
  • Expected Downstream Snaps: Any Snap that accepts document data in its input view. Examples: CSV Parser, HDFS Writer, File Writer.


The user executing the Snap must have Read permissions on the concerned Hadoop directory.

Configuring Accounts

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See Configuring Hadoop Accounts for information on setting up this type of account.

Configuring Views


This Snap has at most one document input view.
OutputThis Snap has exactly one binary output view.
ErrorThis Snap has at most one document error view.


None at this time.

Limitations and Known Issues

None at this time.


Snap Settings

LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.

The URL for the data source (directory). The Snap supports both HFDS and ABFS(S) protocols.

Syntax for a typical HDFS URL:

Syntax for a typical ABFS and an ABFSS URL:

When you use the ABFS protocol to connect to an endpoint, the account name and endpoint details provided in the URL override the corresponding values in the Account Settings fields.

Default value: [None]

File FilterUse glob patterns to display a list of directories or files when you click the Suggest icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with "/", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.

Default Value: *

For more information on glob patterns, click the link below.

 Glob Pattern Interpretation Rules

Glob Pattern Interpretation Rules

The following rules are used to interpret glob patterns:

  • The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period.

  • The ** characters match zero or more characters across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory.

  • The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension.

  • The \ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\ matches a single backslash, and \{ matches a left brace, for example.

  • The ! character is used to exclude matching files from the output. 
  • The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'.

    Within a bracket expression, the '*', '?', and '\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating.

  • The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'.

  • Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name ".login".

  • All other characters match themselves.


  • '*.csv' matches all files with a csv extension in the current directory only.
  • '**.csv' matches all files with a csv extension in the current directory and in all its subdirectories.
  • *[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP.

The relative path and name of the file that must be read.


  • sample.csv
  • tmp/another.csv
  • $filename

Default value:  [None]

User Impersonation

Select this check box to enable user impersonation.

For encryption zones, use user impersonation. 

Default value:  Not selected

For more information on working with user impersonation, click the link below.

 User Impersonation Details

Generic User Impersonation Behavior

When the User Impersonation check box is selected, and Kerberos is the account type, the Client Principal configured in the Kerberos account impersonates the pipeline user.

When the User Impersonation option is selected, and Kerberos is not the account type, the user executing the pipeline is impersonated to perform HDFS Operations. For example, if the user logged into the SnapLogic platform is operator@snaplogic.com, the user name "operator" is used to proxy the super user. 

User impersonation behavior on pipelines running on Groundplex with a Kerberos account configured in the Snap

  • When the User Impersonation checkbox is selected in the Snap, it is the pipeline user who performs the file operation. For example, if the user logged into the SnapLogic platform is operator@snaplogic.com, the user name "operator" is used to proxy the super user.
  • When the User Impersonation checkbox is not selected in the Snap, the Client Principal configured in the Kerberos account performs the file operation.

For non-Kerberised clusters, you must activate Superuser access in the Configuration settings.

HDFS Snaps support the following accounts:

  • Azure storage account
  • Azure Data Lake account
  • Kerberos account
  • No account

When an account is configured with an HDFS Snap, user impersonation settings have no impact on all accounts, except the Kerberos account.

Prevent URL EncodingSelect this checkbox to prevent the Snap from automatically encoding the URL file path (including the query string if it exists) and use the file path value as-is.  

Deselect this checkbox to encode the URLs. The following are some of the common characters that are automatically encoded by the Snap:

Character name Character  URL Encoded value
backslash      \ %5C
Pound #    %23
 percent    %     %25 

And these are some of the characters that are not automatically encoded by the Snap:

Character name Character  URL Encoded value




question mark     



forward slash      






















Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

The binary document header content-location of the HDFS ZipFile Writer input is the name within the ZIP file. (Example: foo.txt). The Snap does not include the 'base directory'. It could contain subdirectories though. On the other hand, the binary document header content-location of the output of the HDFS ZipFile Reader is the name of the ZIP file, the base directory, and the content location provided to the writer. Thus, while each Snap works well independent of each other, it's currently not possible to have a Reader > Writer > Reader combination in a pipeline without using other intermediate Snaps to provide the binary document header information.


Writing and Reading a ZIP File in HDFS

The first part of this example demonstrates how you can use the HDFS ZipFile Write Snap to zip and write a new file into HDFS. The second part of this example demonstrates how you can unzip and check the contents of the newly-created ZIP file.

Click here to download this pipeline. You can also downloaded this pipeline from the Downloads section below.

 Understanding the Sample Pipeline

Create the pipeline as shown below:

The Hadoop Directory Browser Snap

Use a Hadoop Directory Browser Snap to first check the contents of the target directory. This will help you check whether the new file got added to the HDFS directory as expected, later in the example.

Enter the Directory URL as appropriate and specify the File filter as *.zip. This instructs the Snap to list out all the ZIP files in the target directory.

If the Snap executes as expected, you should see the contents of your target directory, as shown below:

Generating a File for Upload

You now need to choose a file to upload into the target directory. You could either select a file directly or use a JSON Generator Snap coupled with a JSON Formatter Snap, as in the example pipeline.

The HDFS ZipFile Writer Snap

Your file is now ready. Configure the HDFS ZipFile Writer Snap to upload the file as a ZIP file into the target directory in HDFS, as shown below.

The Hadoop Directory Browser Snap

Use a Copy Snap to perform two tasks after the ZIP file is created: first, to check whether the new file was created as expected and second, to try and read the contents to the newly-created ZIP file from the target HDFS directory.

To check whether the new file was created, add an HDFS Directory Browser Snap to the pipeline.

If the ZIP file was created, you should see it in the output, as shown below:

HDFS ZipFile Reader

Once you have confirmed that the new ZIP file has been created, use the HDFS ZipFile Reader Snap to read the new ZIP file. If the contents of the new ZIP file is the same as the contents of the input file, you know that the pipeline works!

To read the output of the HDFS ZipFile Read Snap, use a File Reader Snap:

If the contents of the new file is the same as the contents of the original file, you know the example works.

Click here to download this Pipeline. You can also downloaded this pipeline from the Downloads section below.


Writing to S3 files with HDFS version CDH 5.8 or later

When running HDFS version later than CDH 5.8, the Hadoop Snap Pack may fail to write to S3 files. To overcome this, make the following changes in the Cloudera manager:

  1. Go to HDFS configuration.
  2. In Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml, add an entry with the following details:
    • Name: fs.s3a.threads.max
    • Value: 15
  3. Click Save.
  4. Restart all the nodes.
  5. Under Restart Stale Services, select Re-deploy client configuration.
  6. Click Restart Now.


Important steps to successfully reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide Pipeline parameters as applicable.

  File Modified

File HDFS_ZIPFILE_Read_Write.slp

Oct 19, 2018 by Rakesh Chaudhary

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
November 2022main18944 Stable

The AWS S3 and S3 Dynamic accounts now support a maximum session duration of an IAM role defined in AWS.

August 2022main17386 StableExtended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
4.29 Patch429patches16630 Latest
  • Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support AWS Security Token Service (STS) using temporary credentials.
  • Fixed an issue in the following Snaps that use AWS S3 dynamic account, where the Snaps displayed the security credentials like Access Key, Secret Key, and Security Token in the logs. Now, the security credentials in the logs are blurred for the Snaps that use AWS S3 dynamic account.
4.29main15993 Stable

Enhanced the AWS S3 Account for Hadoop account to include the S3 Region field that allows cross-region or proxied cross-region access to S3 buckets in the Parquet Reader and Parquet Writer Snaps.

4.28 Patch428patches15216 LatestAdded the AWS S3 Dynamic account for Parquet Reader and Parquet Writer Snaps.
4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches13769 Latest

Fixed an issue with the Hadoop Directory Browser Snap where the Snap was not listing the files in the given directory for Windows VM.

4.27 Patch427patches12999 LatestEnhanced the Parquet Reader Snap with int96 As Timestamp checkbox, which when selected enables the Date Time Format field. You can use this field to specify a date-time format of your choice for int96 data-type fields. The int96 As Timestamp checkbox is available only when you deselect Use old data format checkbox.





Enhanced the Parquet Writer and Parquet Reader Snaps with Azure SAS URI properties, and Azure Storage Account for Hadoop with SAS URI Auth Type. This enables the Snaps to consider SAS URI given in the settings if the SAS URI is selected in the Auth Type during account configuration. 

4.26426patches12288 Latest

Fixed a memory leak issue when using HDFS protocol in Hadoop Snaps.

4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches9975 Latest

Fixed the dependency issue in Hadoop Parquet Reader Snap while reading from AWS S3. The issue is caused due to conflicting definitions for some of the AWS classes (dependencies) in the classpath.

  • Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
    • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
    • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
4.24 Patch424patches9262 Latest

Enhanced the AWS S3 Account for Hadoop to support role-based access when you select IAM role checkbox.

4.24 Patch424patches8876



Fixes the missing library error in Hadoop Snap Pack when running Hadoop Pipelines in JDK11 runtime.

StableUpgraded with the latest SnapLogic Platform release.
4.23 Patch423patches7440 Latest

Fixes the issue in HDFS Reader Snap by supporting to read and write files larger than 2GB using ABFS(S) protocol.

StableUpgraded with the latest SnapLogic Platform release.
StableUpgraded with the latest SnapLogic Platform release.
4.21 Patchhadoop8853 Latest

Updates the Parquet Writer and Parquet Reader Snaps to support the yyyy-MM-dd format for the DATE logical type.



StableUpgraded with the latest SnapLogic Platform release.
4.20 Patchhadoop8776 Latest

Updates the Hadoop Snap Pack to use the latest version of org.xerial.snappy:snappy-java for compression type Snappy, in order to resolve the java.lang.UnsatisfiedLinkError: org.xerial.snappy.SnappyNative.maxCompressedLength(I)I error.

StableUpgraded with the latest SnapLogic Platform release.
4.19 Patchhadoop8270 Latest

Fixes an issue with the Hadoop Parquet Writer Snap wherein the Snap throws an exception when the input document includes one or all of the following:

  • Empty lists.
  • Lists with all null values.
  • Maps with all null values.
StableUpgraded with the latest SnapLogic Platform release.
4.18 Patchhadoop8033 Latest

Fixed an issue with the Parquet Writer Snap wherein the Snap throws an error when working with WASB protocol.


Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.


Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.


Added a new property, Output for each file written, to handle multiple binary input data in the HDFS Writer Snap.

  • Added two new Snaps: HDFS ZipFile Reader and HDFS ZipFile Writer.
  • Added support for the Data Catalog Snaps in Parquet Reader and Parquet Writer Snaps.
4.14 Patchhadoop5888 Latest

Fixed an issue wherein the Hadoop snaps were throwing an exception when a Kerberized account is provided, but the snap is run in a non-kerberized environment.

  • Added the Hadoop Directory Browser Snap, which browses a given directory path in the Hadoop file system using the HDFS protocol and generates a list of all the files in the directory. It also lists subdirectories and their contents.
  • Added support for S3 file protocol in the ORC Reader, and ORC Writer Snaps.
  • Added support for reading nested schema in the Parquet Reader Snap.
4.13 Patchhadoop5318 Latest
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein Hadoop configuration information does not parse from the client's configuration files.
  • Fixed the HDFS Reader/Writer and Parquet Reader/Writer Snaps, wherein User Impersonation does not work on Hadooplex.


  • KMS encryption support added to AWS S3 account in the Hadoop Snap Pack.
  • Enhanced the Parquet Reader, Parquet Writer, HDFS Reader, and HDFS Writer Snaps to support WASB and ADLS file protocols.
  • Added the AWS S3 account support to the Parquet Reader and Writer Snaps. 
  • Added second input view to the Parquet Reader Snap that when enabled, accepts table schema.
  • Supported with AWS S3, Azure Data Lake, and Azure Storage Accounts.
4.12 Patchhadoop5132 Latest

Fixed an issue with the HDFS Reader Snap wherein the pipeline becomes stale while writing to the output view.



StableUpgraded with the latest SnapLogic Platform release.
4.11 Patchhadoop4275

Addressed an issue with Parquet Reader Snap leaking file descriptors (connections to HDFS data nodes). The Open File descriptor values work stable now,


Added Kerberos support to the standard mode Parquet Reader and Parquet Writer Snaps.

4.10 Patchhadoop4001 Latest

Supported HDFS Writer to write to the encryption zone.

4.10 Patchhadoop3887 Latest

Addressed the suggest issue for the HDFS Reader on Hadooplex.

4.10 Patchhadoop3851 Latest
  • ORC supports read/write from local file system.
  • Addressed an issue to bind the Hive Metadata to Parquet Writer Schema at Runtime.
4.10 Patchhadoop3838 Latest

Made HDFS Snaps work with Zone encrypted HDFS.



  • Updated the Parquet Writer Snap with Partition by property to support the data written into HDFS based on the partition definition in the schema in Standard mode.
  • Support for S3 accounts with IAM Roles added to Parquet Reader and Parquet Writer
  • HDFS Reader/Writer with Kerberos support on Groundplex (including user impersonation).
4.9 Patchhadoop3339 Latest

Addressed the following issues:

  • ORC Reader passing, but ORC Writer failing when run on a Cloudplex.
  • ORC Reader Snap is not routing error to error view.
  • Intermittent failures with the ORC Writer
4.9.0 Patchhadoop3020 Latest

Added missing dependency org.iq80.snappy:snappy to Hadoop Snap Pack.

StableUpgraded with the latest SnapLogic Platform release.



Snap-aware error handling policy enabled for Spark mode in Sequence Formatter and Sequence Parser. This ensures the error handling specified on the Snap is used.

4.7.0 Patchhadoop2343 Latest

Spark Validation: Resolved an issue with validation failing when setting the output file permissions.



  • Updated the HDFS Writer and HDFS Reader Snaps with Azure Data Lake account for standard mode pipelines.
  • HDFS Writer: Spark mode support added to write to a specified directory in an Azure Storage Layer using the wasb file system protocol.
  • HDFS Reader: Spark mode support added to read a single file or an HDFS directory from an Azure Storage Layer.
  • The following Snaps now support error view in Spark mode: HDFS Reader, Sequence Parser.
  • Resolved an issue in HDFS Writer Snap that sends the same data in output & error view.


  • HDFS Reader and HDFS Writer Snaps updated to support IAM Roles for Amazon EC2.
  • Support for Spark mode added to Parquet Reader, Parquet Writer
  • The HBase Snaps are no longer available as of this release.
  • Resolved an issue with Sequence Formatter not working in Spark mode.
  • Resolved an issue with HDFSReader not using the filter set when configuring SparkExec paths.
  • NEW! Parquet Reader and Writer Snaps
  • NEW! ORC Reader and Writer Snaps
  • Spark support added to the HDFS Reader, HDFS Writer, Sequence Formatter, and Sequence Parser Snaps.
  • Behavior change: HDFS Writer in SnapReduce mode now requires the File property to be blank.
  • Implemented wasbs:// protocol support in Hadoop Snap Pack.
  • Resolved an issue with HDFS Reader unable to read all files under a folder (including all files under its subfolders) using the ** filter.