Snap type: | Read |
|---|
Description: | This Snap polls the target directory and looks for files matching the specified pattern. It continues polling at the intervals specified in the Polling interval property until the timeout (as specified in the Polling timeout property). The Snap polls the target directory only; subdirectories, if any, are ignored. Once polling is done, the Snap lists out all files matching the specified pattern. Use the Directory Browser Snap to poll files in the directory and all subdirectories as well as to poll a directory only once. The File Poller Snap can be used where an operation is to be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling timeout value is reached. - Expected upstream Snaps: Any Snap with a document output view, such as Mapper, JSON Generator.
- Expected downstream Snaps: Any Snap with a document input view, such as File Reader, Mapper, JSON Formatter.
- Expected input: An optional document to evaluate expressions in the Directory and/or File filter properties. Please note that each input document will trigger the execution of the Snap.
- Expected output: A full path in each document as a value for a key "path". If there are multiple files matching the filter, the same number of documents will be provided in the output view after each interval.
| Code Block |
|---|
[
{
"path" : "sftp://sftp.smart.com/home/voo/test1.csv"
},
{
"path" : "sftp://sftp.smart.com/home/voo/test2.csv"
}
] |
|
|---|
| Prerequisites: | IAM Roles for Amazon EC2 The 'IAM_CREDENTIAL_FOR_S3' feature is to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain the access rights to the S3 buckets. To enable this feature, the following line should be added to global.properties and the jcc (node) restarted:
jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE Please note this feature is supported in the EC2-type Groundplex only. For more information on IAM Roles, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html |
|---|
| Support and limitations: | ModesLimitation- For S3 folders, the Snap currently supports polling the target directory for a maximum of 10,000 files. If there are more than that, the Snap does not provide any output.
|
|---|
| Account: | This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Binary Account for information on setting up these types of accounts. Account types supported by each protocol are as follows:
| Protocol | Account types |
|---|
| sldb | no account | | s3 | AWS S3 | | ftp | Basic Auth | | sftp | Basic Auth, SSH Auth | | ftps | Basic Auth | | hdfs | no account | | webhdfs | no account | | smb | SMB | | wasb | Azure Storage | | wasbs | Azure Storage | | gs | Google Storage | | file | Local file system |
Required settings for account types are as follows: | Account Type | Settings |
|---|
| Basic Auth | Username, Password | | AWS S3 | Access-key ID, Secret key | | SSH Auth | Username, Private key | | SMB | Domain, Username, Password | | Azure Storage | Account name, Primary access key | | Google Storage | Approval prompt, Application scope, Auto-refresh token
(Read-only properties are Access token, Refresh token, Access token expiration, OAuth2 Endpoint, OAuth2 token and Access type.) |
|
|---|
| Views: |
| Input | This Snap has at most one document input view. |
|---|
| Output | This Snap has exactly one document output view. |
|---|
| Error | This Snap has at most one document error view. |
|---|
|
|---|
Settings |
|---|
Label | Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. |
|---|
Directory
| This property is a URL path to the directory where files will be searched. The expected syntax is: [protocol]://[host][:port]/[path] The supported file protocols are: - s3:
- file:
ftp: - ftps:
- sftp:
- hdfs:
- webhdfs:
sldb: smb: wasb: wasbs: gs:
Example: s3:///[bucket_name]/[dir_path]
sftp://ftp.snaplogic.com:22/home/test/dir
ftp://ftp.snaplogic.com/test/csv
$directory
_directory (A key-value pair with "directory" key should be defined as a pipeline parameter. Ensure that the '=' button is enabled when using parameters.)
file:///D:/testFolder/ (if the Snap is executed in the Windows Groundplex and needs to access the D: drive)
wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/
gs:///testBucket/testDir/
Default value: [None]
| Note |
|---|
- The protocol and the rest of the URL should be separated by "://". The host name and the port number should be between "://" and "/".
- Not all file protocols support "//", use "///" instead. For example, if polling files in SLDB and S3 (see the examples shown above).
- This property should be an absolute path for all protocols except SLDB. For SLDB, the Snap can access only the same project directory or the shared project directory, and cannot access other project directories.
- If you want this property to refer to the SLDB project (or shared project) directory where the pipeline of this Snap belongs to, enter "sldb:///" or leave it blank.
- If the pipeline is created in a project other than the shared project and you want this property to refer to the shared project, enter "shared" or "sldb:///shared".
- If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the SLDB and S3 protocols.
|
|
|---|
File filter | Required. A GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document.
Example: Default value: [None] |
|---|
Polling interval in seconds | Required. The time-gap between each poll request (in seconds). Example: 10 Default value: 30 |
|---|
Polling timeout | Required. A period of time after which file polling must end. You specify the number here and the time unit in the next property. Its unit is selected in the next property. Example: - -1 (to poll indefinitely)
- 0 (to poll once)
- 60 (its unit shown in the next property)
Default value: 30 | Note |
|---|
Configure this property based on the expected number of files in the target directory. If there are many files and this property's value is small, the Snap may complete the operation and stop before the file is found. |
|
|---|
| Polling-timeout unit | Unit for the polling timeout. Allowed values are SECONDS, MINUTES and HOURS. Example: SECONDS Default value: MINUTES |
|---|
