Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Snap type:

Read

Description:

This Snap browses a given directory path from various sources (such as FTP, FTPS, SFTP, SLDB and S3) and lists the attributes (such as Name, Type, Size, Last Modification Time) of the contents (files or sub-folders).

  • Expected upstream Snaps: Upstream Snap can be given. 
  • Expected downstream Snaps: CSV Formatter followed by File Writer. 
  • Expected input: Directory Path to be browsed and File Filter Pattern to be applied. 
  • Expected output: The attributes (name, size, etc) of the contents of the directory path matching the filter pattern.
Prerequisites:

Multiexcerpt include macro
nameEC2Prerequisite
pageFile Reader

Support and limitations:
Known Issues:None.
Account: 

This Snap uses account references created on the Accounts page of the SnapLogic Manager to handle access to this endpoint. 

This Snap supports a Basic auth account, an AWS S3 auth account, SSH Auth account, SMB account, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.

Views:


InputThis Snap has at most one optional document input view. It contains values for the directory path to be browsed and the GLOB filter to be applied to select the contents.
OutputThis Snap has exactly one output view that provides the various attributes (such as Name, Type, Size, Owner, Last Modification Time) of the contents of the given directory path.  Only those contents are selected that match the given GLOB filter.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.


Settings

Label

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Directory 


The URL for the data source (directory). It should start with a protocol. The protocols that are currently supported include:

  • sldb:

  • s3:

  • ftp:

  • ftps:

  • sftp:

  • hdfs:

  • wasb:

  • wasbs:

  • smb:
  • gs:

  • adl:

  • file:

The Directory field can be a JavaScript expression, which is evaluated with the values from the input view document. 

For SLDB: protocol, relative path is accepted. For example, if this field is left blank, the SLDB project folder of the pipeline is selected. Only two directories can be selected in the SLDB: protocol: the current project directory and the "shared" directory. If the pipeline belongs to the Shared project, only "shared" directory can be selected. No subdirectory is supported for SLDB: as of now. 

Note
  • SnapLogic automatically appends "azuredatalakestore.net" to the store name you specify when using Azure Data Lake; therefore, you do not need to add 'azuredatalakestore.net' to the URI while specifying the directory.
  • The FTPS file protocol works only in explicit mode. The implicit mode is not supported.
  • When using the file:/// protocol, file access is based on the user permissions of the associated Snaplex (by default, this is the Snap user). Use file system access with caution, and ensure that the file system is cleaned after use.

Examples

  • adl://storename/folder/filename (to read the file from a location of the storage)
  • file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access D: drive)
  • ftp://ftp.mock.com/vehicles

  • gs:///testBucket/testDir/ (if the bucket name is 'testBucket')
  • project-name
  • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

    For region names and their details, see AWS Regions and Endpoints.

    Note

    Release 4.8.0 does not support the London region eu-west-2.


  • sftp://sftp.mock.com/students/grades/
  • share
  • sldb:///shared
  • sldb:///project-name

  • smb://server/share/folder/ (if the target folder is /share/folder in server)
  • wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/ (if the name of the container is 'Snaplogic')

Default value: [None]

File filter

The GLOB pattern can be applied to select the contents (files/sub-folders) of the directory. You cannot recursively navigate through the directory structures.

The File filter property can be a JavaScript expression, which will be evaluated with the values from the input view document.

Example:

  • *.txt

  • ab????xx.*x

  • *.[jJ][sS][oO][nN](as of the May 29th, 2015 release)

Default value: [None]

Ignore empty result

If selected, no document will be written to the output view when the result is empty. If this property is not selected and the Snap has received an input document, the input document will be passed through to the output view. If this property is not selected and there is no input document, an empty document will be written to the output view.
Default value: Selected

Display update time and owner for S3 subdirectories

Applicable only for S3 directory.

If selected, the Snap produces Update time and Owner attributes in the output documents for S3 subdirectories. However, it may take a long time or even freeze if there are very large number of files and subdirectories in the file structure under the specified S3 directory. This is because the Snap has to completely examine the S3 objects under the directory recursively in order to get Update time and Owner attributes for subdirectories.

If not selected, the Snap does not produce Update time and Owner attributes in the output documents for S3 subdirectories. Snap output is faster since it examines only S3 files and subdirectories immediately under the specified S3 directory.

Default value: Selected

Number of retries

Specifies the maximum number of retry attempts in case of a network failure.

Example:  3

Minimum value: 0

Default value: 0

Note
titleNumber_of_Retries

Whenever the Snap needs to connect to an endpoint and the Connection object is either null or closed, the Snap opens the connection and tries to connect thrice before aborting.

During each retry, the Snap makes a fresh attempt to download and populate the file attributes list into a temporary local file. 

  • Upon successful download, the Snap streams the entire data from the temporary file to the downstream Snaps and deletes the file. 
  • In case of an error, the Snap waits for the duration specified in the Retry interval field and attempts a fresh download of file attributes list. 


Retry interval (seconds)

Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Example:  3

Minimum value: 1

Default value: 1

Advanced properties
Use this field set to define specific settings for directory browsing. Click  to add a new row for defining an advanced property. This field set comprises the following fields:
  • Properties
  • Values
Properties

This list contains the following fields:

  • SAS URI
  • Use fast file-type check for wasb/wasbs protocols
SAS URI


Multiexcerpt macro
nameSASURI_Description_NoAccountSAS

Specify the URI of the Shared Access Storage (SAS) to be accessed. Click  to add the SAS URI. You can generate the SAS URI either from the  Snap or from the Azure portal → Shared access signature.

The supported SAS types are:

  • Service SAS on a container
  • Service SAS on blob
If you provide SAS URI in the Snap, the Primary access key given in the account settings is overridden while authentication. If you do not provide the SAS URI, the Snap considers the Primary access key in the account settings.

Example: https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-
29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60
-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D

Default value: N/A

Ensure that the URI is specified in the format described here.


SAS URI provided in the Snap will completely lead to suppressing the settings provided in the account. SAS URI from the Snaps will be strictly used.




Multiexcerpt macro
nameFast_File_Type_Check_Name
Use fast file-type check for wasb/wasbs protocols



Multiexcerpt macro
nameFast_File_Type_Check_Desc

Specifies whether the Snap can use a fast file-type check while browsing through a given file/directory path (for WASB and WASBS file protocols).

Set this field of the Snap to true to save execution times when working with WASB and WASBS file protocols.

Default value: N/A

Snap execution results may not be accurate when using this field. Set this field to true if longer execution times are not a constraint.



Multiexcerpt include macro
nameSnap Execution
pageAnaplan Read


Multiexcerpt include macro
nameSnap_Execution_Introduced
pageAnaplan Read


Excerpt Include
Binary Snap Pack
Binary Snap Pack
nopaneltrue