This Snap browses a given directory path from various sources (such as FTP, FTPS, SFTP, SLDB and S3) and lists the attributes (such as Name, Type, Size, Last Modification Time) of the contents (files or sub-folders).
Expected upstream Snaps: Upstream Snap can be given.
Expected downstream Snaps: CSV Formatter followed by File Writer.
Expected input: Directory Path to be browsed and File Filter Pattern to be applied.
Expected output: The attributes (name, size, etc) of the contents of the directory path matching the filter pattern.
This Snap uses account references created on the Accounts page of the SnapLogic Manager to handle access to this endpoint.
This Snap supports a Basic auth account, an AWS S3 auth account, SSH Auth account, SMB account, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.
Views:
Input
This Snap has at most one optional document input view. It contains values for the directory path to be browsed and the GLOB filter to be applied to select the contents.
Output
This Snap has exactly one output view that provides the various attributes (such as Name, Type, Size, Owner, Last Modification Time) of the contents of the given directory path. Only those contents are selected that match the given GLOB filter.
Error
This Snap has at most one document error view and produces zero or more documents in the view.
Settings
Label
Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.
Directory
The URL for the data source (directory). It should start with a protocol. The protocols that are currently supported include:
sldb:
s3:
ftp:
ftps:
sftp:
hdfs:
wasb:
wasbs:
smb:
gs:
adl:
file:
The Directory field can be a JavaScript expression, which is evaluated with the values from the input view document.
For SLDB: protocol, relative path is accepted. For example, if this field is left blank, the SLDB project folder of the pipeline is selected. Only two directories can be selected in the SLDB: protocol: the current project directory and the "shared" directory. If the pipeline belongs to the Shared project, only "shared" directory can be selected. No subdirectory is supported for SLDB: as of now.
Note
SnapLogic automatically appends "azuredatalakestore.net" to the store name you specify when using Azure Data Lake; therefore, you do not need to add 'azuredatalakestore.net' to the URI while specifying the directory.
The FTPS file protocol works only in explicit mode. The implicit mode is not supported.
When using the file:/// protocol, file access is based on the user permissions of the associated Snaplex (by default, this is the Snap user). Use file system access with caution, and ensure that the file system is cleaned after use.
Examples:
adl://storename/folder/filename(to read the file from a location of the storage)
file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access D: drive)
ftp://ftp.mock.com/vehicles
gs:///testBucket/testDir/ (if the bucket name is 'testBucket')
Release 4.8.0 does not support the London region eu-west-2.
sftp://sftp.mock.com/students/grades/
share
sldb:///shared
sldb:///project-name
smb://server/share/folder/ (if the target folder is /share/folder in server)
wasb:///Snaplogic/testDir/orwasbs:///Snaplogic/testDir/ (if the name of the container is 'Snaplogic')
Default value: [None]
File filter
The GLOB pattern can be applied to select the contents (files/sub-folders) of the directory. You cannot recursively navigate through the directory structures.
The File filter property can be a JavaScript expression, which will be evaluated with the values from the input view document.
Example:
*.txt
ab????xx.*x
*.[jJ][sS][oO][nN](as of the May 29th, 2015 release)
Default value: [None]
Ignore empty result
If selected, no document will be written to the output view when the result is empty. If this property is not selected and the Snap has received an input document, the input document will be passed through to the output view. If this property is not selected and there is no input document, an empty document will be written to the output view. Default value: Selected
Display update time and owner for S3 subdirectories
Applicable only for S3 directory.
If selected, the Snap producesUpdate timeandOwnerattributes in the output documents for S3 subdirectories. However, it may take a long time or even freeze if there are very large number of files and subdirectories in the file structure under the specified S3 directory. This is because the Snap has to completely examine the S3 objects under the directory recursively in order to getUpdate time andOwnerattributes for subdirectories.
If not selected, the Snap does not produceUpdate timeandOwnerattributes in the output documents for S3 subdirectories. Snap output is faster since it examines only S3 files and subdirectories immediately under the specified S3 directory.
Default value: Selected
Number of retries
Specifies the maximum number of retry attempts in case of a network failure.
Example: 3
Minimumvalue: 0
Default value: 0
Note
title
Number_of_Retries
Whenever the Snap needs to connect to an endpoint and the Connection object is either null or closed, the Snap opens the connection and tries to connect thrice before aborting.
During each retry, the Snap makes a fresh attempt to download and populate the file attributes list into a temporary local file.
Upon successful download, the Snap streams the entire data from the temporary file to the downstream Snaps and deletes the file.
In case of an error, the Snap waits for the duration specified in the Retry interval field and attempts a fresh download of file attributes list.
Retry interval (seconds)
Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.
Example: 3
Minimum value: 1
Default value: 1
Advanced properties
Use this
fieldset
field set to define specific settings for directory browsing.
Click +
Click Image Addedto add a new row for defining an advanced property. This
fieldset comprises
field set comprises the following fields:
Properties
Values
Properties
This list contains the following fields:
SAS URI
Use fast file-type check for wasb/wasbs protocols
SAS URI
Multiexcerpt macro
name
SASURI_Description_NoAccountSAS
Specifies Specify the URI of the Shared Access Storage (SAS) to be accessed. Click '+' Click to add the SAS URI. You can generate the SAS URI either from the Snap or from the Azure portal → Shared access signature.
Supported The supported SAS types are:
Service SAS on a container
Service SAS on blob
If you provide SAS URI is given in the Snapoverrides , the Primary access key given in the in the account settings is overridden while authentication. If you do not provide the SAS URI in URI, the Snap settings is empty, then considers the Primary access key from in the account settingsis considered.
to add a new row for defining an advanced property. This