/
Directory Browser

Directory Browser

In this article

Overview

  • This Snap browses a given directory path from various sources (such as FTP, FTPS, SFTP, SLDB and S3) and lists the attributes (such as Name, Type, Size, Last Modification Time) of the contents (files or sub-folders).

Important

We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps.

Learn more: Migration from Binary Snaps to Amazon S3 Snaps.

Snap Type

The Directory Browser Snap is a Read-type Snap.

Prerequisites

IAM Roles for Amazon EC2

The 'IAM_CREDENTIAL_FOR_S3' feature is used to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain access rights to the S3 buckets. To enable this feature, set the Global properties (Key-Value parameters) and restart the JCC:
jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

This feature is supported in the EC2-type Groundplex only. Learn more.  


Connect to FTP server:

To connect to the FTP server that needs to reuse the session for data transfer over TLS protocol, add:

-DFTPS_SSL_TLS_PROTOCOL=TLSV1.2  (or) TLSV1.3property as a JVM option under the Global properties of the Node Properties tab:

Support for Ultra Pipelines 

Works in Ultra Pipelines

Limitations

None.

Known Issues

  • This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex. 

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 0

  • Max: 1

  • Mapper

  • S3 Copy

Directory Path to be browsed and File Filter Pattern to be applied. It contains values for the directory path to be browsed and the GLOB filter to be applied to select the contents.

Output

Document


  • Min: 1

  • Max: 1

  • CSV Formatter

  • File Writer

The attributes (name, size, and so on) of the contents of the directory path matching the filter pattern. This Snap provides the various attributes (such as Name, Type, Size, Owner, Last Modification Time) of the contents of the given directory path.  Only those contents are selected that match the given GLOB filter.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk (*): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon (  ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon (): Indicates that you can add fields in the fieldset.

  • Remove icon (): Indicates that you can remove fields from the fieldset.

  • Upload icon (): Indicates that you can upload a file.

Field

Field Type

Description

Label*


Default Value: Directory Browser
ExampleDirectory Browser

String

Specify a unique name for the Snap.

Directory 

Default Value: N/A

Examples

  • Read the file from a location of the storage: adl://storename/folder/filename 

  • If the Snap is executed in the Windows Groundplex and needs to access D: drive): file:///D:/testFolder/ 

  • ftp://ftp.mock.com/vehicles

  • If the bucket name is 'testBucket': gs:///testBucket/testDir/ 

  • project-name

  • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

  • File in VPC: s3:///snaplogic-bucket@bucket.vpce-0e340d8d5b0eae90a-k9wbom46-us-west-2a.s3.us-west2.vpce.amazonaws.com

  • sftp://sftp.mock.com/students/grades/

  • share

  • sldb:///shared

  • sldb:///project-name

  • If the target folder is /share/folder in server: smb://server/share/folder/ 

  • If the name of the container is 'Snaplogic': wasb:///Snaplogic/testDir/
    or wasbs:///Snaplogic/testDir/ 

String/Expression

Specify the URL for the data source (directory) that should begin with a protocol. Following are the protocols that are currently supported:

  • adls  Gen 2 Storage

  • adl:

  • file:

  • ftp:
  • ftps:

  • sftp:

  • hdfs:

  • smb:
  • gs:

  • sldb:

  • s3:

  • wasb:

  • wasbs:

File filter

Default Value: N/A
Example:

  • *.txt

  • ab????xx.*x

  • *.[jJ][sS][oO][nN]

String/Expression

Specify the Glob pattern for filtering the files. The Glob pattern can be applied to select the contents (files/sub-folders) of the directory. You cannot recursively navigate through the directory structures.

Ignore empty result

Default Value: Selected

Checkbox

Select this checkbox to enable the Snap to not write a document to the output view when the result is empty.

Deselect this checkbox in the following cases:

  • If the Snap receives an input document, the input document is passed through to the output view.

  • If the Snap does not receive an input document, an empty document is written to the output view.

Display update time and owner for S3 subdirectories


Default Value: Selected

Checkbox

Applicable only for S3 directory.

If selected, the Snap produces Update time and Owner attributes in the output documents for S3 subdirectories. However, it may take a long time or even freeze if there are very large number of files and subdirectories in the file structure under the specified S3 directory. This is because the Snap has to completely examine the S3 objects under the directory recursively in order to get Update time and Owner attributes for subdirectories.

If not selected, the Snap does not produce Update time and Owner attributes in the output documents for S3 subdirectories. Snap output is faster since it examines only S3 files and subdirectories immediately under the specified S3 directory.

Number of retries


Default Value: 0
Example:  3

String/Expression

Specify the maximum number of retry attempts in case of a network failure.

Minimum value: 0

Retry interval (seconds)


Default Value:1
Example: 3

String/Expression

Specify the minimum number of seconds the Snap must wait before attempting recovery from a network failure.

Minimum value: 1

Advanced properties

Use this field set to define specific settings for directory browsing. 

Properties

SAS URI

Dropdown list



Specify the URI of the Shared Access Storage (SAS) you need to access. You can generate the SAS URI either from the SAS Generator Snap or from the Azure portal → Shared access signature.

The supported SAS types are:

  • Service SAS on a container
  • Service SAS on blob
  • Ensure that the URI is specified in the format described here.
  • If you provide SAS URI in this field, then:
    • the Primary access key given in the account settings is overridden while authentication. If you do not provide the SAS URI, the Snap considers the Primary access key in the account settings.
    • only this URL is used and the Snap ignores the SAS URI settings that you have configured in the associated account.


Default value
: N/A

Example: https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-
29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60
-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D

Use fast file-type check for wasb/wasbs protocols

String/Expression

Specifies whether the Snap can use a fast file-type check while browsing through a given file/directory path (for WASB and WASBS file protocols).

Set this field of the Snap to true to save execution times when working with WASB and WASBS file protocols.

Default value: N/A

Snap execution results may not be accurate when using this field. Set this field to true if longer execution times are not a constraint.


Snap Execution


Default ValueValidate & Execute
Example: Execute only


Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.



 Click to view/expand


Release Snap Pack VersionDateTypeUpdates
November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.
August 2024438patches28197 Latest
  • Fixed an issue in the Binary Snap Pack that caused intermittent errors on the Groundplex nodes when using the SMB protocol. The errors occurred when multiple IPs were returned for the same address, and one of the IPs was unreachable by the SMB server
  • Fixed an issue in the Directory Browser Snap where a mount error was displayed if the hostname contained an underscore, resulting in an empty hostname in the SMB URL.

August 2024

main27765

 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

May 2024437patches27146 LatestEnhanced the Azure Storage Account with Managed Identity, which provides applications with an automatically managed identity for connecting to resources that support Microsoft Entra ID authentication.
May 2024437patches26873 LatestFixed an issue with the File Poller Snap where the Snap applied a case-sensitive filter in the Windows operating system.
May 2024437patches26592 Latest

Enhanced the Decompress Snap to support encrypted and unencrypted ZIP and 7z files through the new File Password Account type.

May 2024main26341 Stable

The Azure Data Lake Account has been removed from the Binary Snap Pack because Microsoft retired the Azure Data Lake Storage Gen1 protocol on February 29, 2024. We recommend replacing your existing Azure Data Lake Account with other Azure Accounts.

February 2024436patches25711 Latest

Fixed an SMB (server message block) connectivity issue within the Binary Snap Pack, as the incorrect name provided by the SMB client did not match the Windows cluster virtual name, affecting the SPN (service principal name) connection.

February 2024436patches25332 Latest

Fixed a null pointer exception in the Public Key Account for Binary Snap Pack when the Public Key field value is null. Now, the account displays a configuration exception for a null value.

February 2024436patches25241 Latest

Fixed an issue with the File Writer Snap, which partially writes the file for some FTPS servers by pausing for seconds before closing the output stream as specified in the global property ftpsDelayBeforeClosing.

February 2024436patches25161 LatestFixed an issue with the File Poller Snap that displayed an exception when an unauthorized character ':' was used in the Windows Snaplex.
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023435patches24525 LatestFixed an issue with the File Poller Snap that caused it to poll recursively for files in the root directory of the S3 bucket.
November 2023435patches23780 Latest

Fixed an issue with the Binary Snaps that failed to build a data connection when connecting to the FTP server over FTPS protocol.

November 2023main23721 Stable

Updated and certified against the current SnapLogic Platform release.

August 2023434patches23502 LatestFixed an issue with the File Operation Snap (SFTP protocol) where the error message did not display during a move operation, although the existing file was available in the target path and the Error if exists checkbox was selected.
August 2023434patches23302 Latest

Fixed an issue that occurred when node properties were used to override default algorithm specifications for SFTP operations.

August 2023434patches22976 LatestFixed an issue that caused account credentials to be visible in the stack trace of some failed SFTP operations.
August 2023434patches22842 Latest

Fixed an issue that caused NTLM authentication issues when trying to access SMB servers.

August 2023