File Poller

Read

IAM Roles for Amazon EC2

The 'IAM_CREDENTIAL_FOR_S3' feature is to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain the access rights to the S3 buckets. To enable this feature, the following line should be added to global.properties and the jcc (node) restarted:
jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

Please note this feature is supported in the EC2-type Groundplex only.

For more information on IAM Roles, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

• Ultra pipelines: May work in Ultra Pipelines.
• Spark mode: Not supported in Spark mode.

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Binary Account for information on setting up these types of accounts. Account types supported by each protocol are as follows:

Required settings for account types are as follows:

This property is a URL path to the directory where files will be searched. The supported file protocols are:

• s3:
• file:

• ftp:

• ftps:
• sftp: 
• hdfs:
• webhdfs:

• sldb: 

• smb:

• wasb:

• wasbs:

• gs:

The property can be a JavaScript expression which will be evaluated with values from the input view document and the pipeline parameters. The property should have the syntax:

        [protocol]://[host][:port]/[path]

Please note "://" is a separator between the file protocol and the rest of the URL and the host name and the port number should be between "://" and "/". If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the sldb and s3 protocols.

This property should be an absolute path for all protocols except sldb. For sldb, the Snap can access only the same project directory or the shared project directory, and cannot access other project directories.

You may leave this property blank to indicate the current sldb project where the pipeline belongs to.

Example: 

• If you want this property to refer to the sldb project (or shared project) directory where the pipeline of this Snap belongs to, enter "sldb:///" or leave it blank.
• If the pipeline is created in a project other than the shared project and you want this property to refer to the shared project, enter "shared" or "sldb:///shared".
• s3:///[bucket_name]/[dir_path]
• sftp://ftp.snaplogic.com:22/home/test/dir
• ftp://ftp.snaplogic.com/test/csv
• $directory (The value of the $directory is obtained from the input document and the document should have an entry with the "directory" key. You must press the '=' button.)
• _directory (A key/value pair with "directory" key should be defined as a pipeline parameter. You must press the '=' button.)
• file:///D:/testFolder/  (if the Snap is executed in the Windows Groundplex and needs to access D: drive)
• wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/ (if the name of the container is 'Snaplogic')
• gs:///testBucket/testDir/ (if the bucket name is 'testBucket')

Default value:  [None]

A GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document.
Example:

• *.txt
• ab????xx.csv

Default value:  [None]

A time interval in seconds to search the directory

Example: 10

Default value: 30

A time to end polling. Its unit is selected in the next property.

Example:

• -1 (to poll indefinitely)
• 0 (to poll once)
• 60 (its unit shown in the next property)

Default value: 30

Unit for polling timeout. Allowed values are SECONDS, MINUTES and HOURS.

Example: SECONDS

Default value: MINUTES