Directory Browser
In this article
Overview
- This Snap browses a given directory path from various sources (such as FTP, FTPS, SFTP, SLDB and S3) and lists the attributes (such as Name, Type, Size, Last Modification Time) of the contents (files or sub-folders).
Important
We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps.
Learn more: Migration from Binary Snaps to Amazon S3 Snaps.
Snap Type
The Directory Browser Snap is a Read-type Snap.
Prerequisites
IAM Roles for Amazon EC2
The 'IAM_CREDENTIAL_FOR_S3' feature is used to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain access rights to the S3 buckets. To enable this feature, set the Global properties (Key-Value parameters) and restart the JCC:jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE
This feature is supported in the EC2-type Groundplex only. Learn more.
Connect to FTP server:
To connect to the FTP server that needs to reuse the session for data transfer over TLS protocol, add:
-DFTPS_SSL_TLS_PROTOCOL=TLSV1.2 (or) TLSV1.3
property as a JVM
option under the Global properties of the Node Properties tab:
Support for Ultra Pipelines
Works in Ultra Pipelines.
Limitations
None.
Known Issues
- This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex.
Snap Views
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document |
|
| Directory Path to be browsed and File Filter Pattern to be applied. It contains values for the directory path to be browsed and the GLOB filter to be applied to select the contents. |
Output | Document |
|
| The attributes (name, size, and so on) of the contents of the directory path matching the filter pattern. This Snap provides the various attributes (such as Name, Type, Size, Owner, Last Modification Time) of the contents of the given directory path. Only those contents are selected that match the given GLOB filter. |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
Snap Settings
Asterisk (*): Indicates a mandatory field.
Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.
Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon (): Indicates that you can add fields in the fieldset.
Remove icon (): Indicates that you can remove fields from the fieldset.
Upload icon (): Indicates that you can upload a file.
Field | Field Type | Description | ||
---|---|---|---|---|
Label*
| String | Specify a unique name for the Snap. | ||
Directory Default Value: N/A Examples:
| String/Expression | Specify the URL for the data source (directory) that should begin with a protocol. Following are the protocols that are currently supported:
| ||
File filter Default Value: N/A
| String/Expression | Specify the Glob pattern for filtering the files. The Glob pattern can be applied to select the contents (files/sub-folders) of the directory. You cannot recursively navigate through the directory structures. | ||
Ignore empty result Default Value: Selected | Checkbox | Select this checkbox to enable the Snap to not write a document to the output view when the result is empty. Deselect this checkbox in the following cases:
| ||
Display update time and owner for S3 subdirectories Default Value: Selected | Checkbox | Applicable only for S3 directory. If selected, the Snap produces Update time and Owner attributes in the output documents for S3 subdirectories. However, it may take a long time or even freeze if there are very large number of files and subdirectories in the file structure under the specified S3 directory. This is because the Snap has to completely examine the S3 objects under the directory recursively in order to get Update time and Owner attributes for subdirectories. If not selected, the Snap does not produce Update time and Owner attributes in the output documents for S3 subdirectories. Snap output is faster since it examines only S3 files and subdirectories immediately under the specified S3 directory. | ||
Number of retries Default Value: 0 | String/Expression | Specify the maximum number of retry attempts in case of a network failure. Minimum value: 0 | ||
Retry interval (seconds)
| String/Expression | Specify the minimum number of seconds the Snap must wait before attempting recovery from a network failure. Minimum value: 1 | ||
Advanced properties | Use this field set to define specific settings for directory browsing. | |||
Properties | SAS URI | Dropdown list | Specify the URI of the Shared Access Storage (SAS) you need to access. You can generate the SAS URI either from the SAS Generator Snap or from the Azure portal → Shared access signature. The supported SAS types are:
Example: https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04- | |
Use fast file-type check for wasb/wasbs protocols | String/Expression | Specifies whether the Snap can use a fast file-type check while browsing through a given file/directory path (for WASB and WASBS file protocols). Set this field of the Snap to true to save execution times when working with WASB and WASBS file protocols. Default value: N/A Snap execution results may not be accurate when using this field. Set this field to true if longer execution times are not a constraint. | ||
Snap Execution
| Dropdown list | Select one of the three modes in which the Snap executes. Available options are:
|