Page Comparison

Snap type:	Read

Description:

This Snap polls

a

the target directory

looking

and looks for files matching the specified pattern. It continues polling at the intervals specified in the Polling interval property until the timeout (as specified in the Polling timeout property). The Snap polls the target directory only; subdirectories, if any, are ignored. Once polling is done, the Snap lists out all files matching the specified pattern. Use the Directory Browser Snap to poll files in the directory and all subdirectories as well as to poll a directory only once.

The File Poller Snap can be used where an operation is to be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling timeout value is reached.

Expected upstream Snaps: Any Snap with a document output view, such as Mapper, JSON Generator.
Expected downstream Snaps: Any Snap with a document input view, such as File Reader, Mapper, JSON Formatter.
Expected input: An optional document to evaluate expressions in the Directory and/or File filter properties. Please note that each input document will trigger the execution of the Snap.
Expected output: A full path in each document as a value for a key "path". If there are multiple files matching the filter, the same number of documents will be provided in the output view after each interval.

Code Block
[ { "path" : "sftp://sftp.smart.com/home/voo/test1.csv" }, { "path" : "sftp://sftp.smart.com/home/voo/test2.csv" } ]

Prerequisites:

IAM Roles for Amazon EC2

The 'IAM_CREDENTIAL_FOR_S3' feature is to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain the access rights to the S3 buckets. To enable this feature, the following line should be added to global.properties and the jcc (node) restarted:
jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

Please note this feature is supported in the EC2-type Groundplex only.

For more information on IAM Roles, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

Support and limitations:

Ultra pipelines: May work in Ultra Pipelines.
Spark mode

(Deprecated)

: Not supported in Spark mode.


Account:	This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Binary Account for information on setting up these types of accounts. Account types supported by each protocol are as follows:

Protocol	Account types
sldb	no account
s3	AWS S3
ftp	Basic Auth
sftp	Basic Auth, SSH Auth
ftps	Basic Auth
hdfs	no account
webhdfs	no account
smb	SMB
wasb	Azure Storage
wasbs	Azure Storage
gs	Google Storage
file	Local file system

Required settings for account types are as follows:

Account Type	Settings
Basic Auth	Username, Password
AWS S3	Access-key ID, Secret key
SSH Auth	Username, Private key
SMB	Domain, Username, Password
Azure Storage	Account name, Primary access key
Google Storage	Approval prompt, Application scope, Auto-refresh token (Read-only properties are Access token, Refresh token, Access token expiration, OAuth2 Endpoint, OAuth2 token and Access type.)

Views:

Input	This Snap has at most one document input view.
Output	This Snap has exactly one document output view.
Error	This Snap has at most one document error

view and produces zero or more documents in the view. If the Snap fails during the operation, an error document is sent to the error

view

containing the fields error, reason, resolution, and stacktrace

.

Settings

Label

Directory
	Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Please note "://" is a separator between the file

This property is a URL path to the directory where files will be searched. The expected syntax is:

[protocol]://[host][:port]/[path]

The supported file protocols are:

s3:
file:
ftp:
ftps:
sftp:
hdfs:
webhdfs:
sldb:
smb:
wasb:
wasbs:
gs:

The property can be a JavaScript expression which will be evaluated with values from the input view document and the pipeline parameters. The property should have the syntax:

[protocol]://[host][:port]/[path]

Example:

- s3:///[bucket_name]/[dir_path]
- sftp://ftp.snaplogic.com:22/home/test/dir
- ftp://ftp.snaplogic.com/test/csv
- $directory
- _directory (A key-value pair with "directory" key should be defined as a pipeline parameter. Ensure that the '=' button is enabled when using parameters.)
- file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access the D: drive)
- wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/
- gs:///testBucket/testDir/

Default value: [None]

Note
The protocol and the rest of the URL

and the

should be separated by "://". The host name and the port number should be between "://" and "/".

If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the sldb and s3 protocols.

Not all file protocols support "//", use "///" instead. For example, if polling files in SLDB and S3 (see the examples shown above).
This property should be an absolute path for all protocols

except sldb

except SLDB.

For sldb

Example:

For SLDB, the Snap can access only the same project directory or the shared project directory, and cannot access other project directories.

You may leave this property blank to indicate the current sldb project where the pipeline belongs to.

If you want this property to refer to

the sldb project

Default value: [None]

the SLDB project (or shared project) directory where the pipeline of this Snap belongs to, enter "sldb:///" or leave it blank.
If the pipeline is created in a project other than the shared project and you want this property to refer to the shared project, enter "shared" or "sldb:///shared".

s3:///[bucket_name]/[dir_path]

sftp://ftp.snaplogic.com:22/home/test/dir

ftp://ftp.snaplogic.com/test/csv

$directory (The value of the $directory is obtained from the input document and the document should have an entry with the "directory" key. You must press the '=' button.)

_directory (A key/value pair with "directory" key should be defined as a pipeline parameter. You must press the '=' button.)

file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access D: drive)

wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/ (if the name of the container is 'Snaplogic')

gs:///testBucket/testDir/ (if the bucket name is 'testBucket')

If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the SLDB and S3 protocols.

File filter

Required.

A GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document.
Example:

*.txt
ab????xx.csv

Default value: [None]


Polling interval in seconds	Required.

A time interval in seconds to search the directory

The time-gap between each poll request (in seconds).

Example: 10

Default value: 30


Polling timeout	Required.

A

time to end polling

period of time after which file polling must end. You specify the number here and the time unit in the next property. Its unit is selected in the next property.

Example:

-1 (to poll indefinitely)
0 (to poll once)
60 (its unit shown in the next property)

Default value: 30

Note
Configure this property based on the expected number of files in the target directory. If there are many files and this property's value is small, the Snap may complete the operation and stop before the file is found.

Polling-timeout unit

Unit for

polling

the polling timeout. Allowed values are SECONDS, MINUTES and HOURS.

Example: SECONDS

Default value: MINUTES

Examples

Insert excerpt

	Binary Snap Pack
	Binary Snap Pack
nopanel	true

Versions Compared

Old Version 8

New Version 9

Key

Settings

Examples