On this Page
Table of Contents | ||||
---|---|---|---|---|
|
Snap type: | Read | |||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Description: | This Snap polls the target directory and looks for file names matching the specified pattern. It continues polling at the intervals specified in the Polling interval property until the timeout (specified in the Polling timeout property) is reached. Once polling is done, the Snap lists all files whose names match the specified pattern.
The File Poller Snap can be used in situations where an operation must be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling interval value is reached.
| |||||||||||||||||||||||||||||||||||||||||
Prerequisites: |
| |||||||||||||||||||||||||||||||||||||||||
Support and limitations: |
| |||||||||||||||||||||||||||||||||||||||||
Known Issues | The Snap is expected to fail if there is no account selected. However, the Snap may execute successfully without any account if all the following conditions exist:
| |||||||||||||||||||||||||||||||||||||||||
Behavior Change | The File Poller Snap now honors the value specified in the Polling timeout field instead of polling indefinitely in case of poor file polling operations. To handle indefinite polling operations the polling is done in a separate thread. However, when the execution time exceeds the value specified in the Polling timeout, a timeout exception is written to the log to prevent the polling from getting stuck and the Snap continues polling depending on the Polling timeout.
| |||||||||||||||||||||||||||||||||||||||||
Account: | This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap. Account types supported by each protocol are as follows:
Required settings for account types are as follows:
| |||||||||||||||||||||||||||||||||||||||||
Views: |
| |||||||||||||||||||||||||||||||||||||||||
Settings | ||||||||||||||||||||||||||||||||||||||||||
Label | Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | |||||||||||||||||||||||||||||||||||||||||
Directory | This property is a URL path to the directory where files will be searched. The expected syntax is: The supported file protocols are:
Example:
Default value: [None]
| |||||||||||||||||||||||||||||||||||||||||
File filter | Required. A GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document.
Default value: [None]
| |||||||||||||||||||||||||||||||||||||||||
Polling interval in seconds | Required. The time-gap between each poll request (in seconds). Example: 10 Default value: 30 | |||||||||||||||||||||||||||||||||||||||||
Polling timeout | Required. A period of time after which file polling must end. Specify a value for polling timeout in this field and its unit in the subsequent field. Example:
Default value: 30
| |||||||||||||||||||||||||||||||||||||||||
Polling-timeout unit | Unit for the polling timeout. Allowed values are SECONDS, MINUTES and HOURS. Example: SECONDS Default value: MINUTES | |||||||||||||||||||||||||||||||||||||||||
Only Output on Change | Select this check box to instruct the Snap to provide an output only when there is a change in the contents of the polled directory. When selected, the Snap provides an output during its initial run if it finds matching documents. However, it provides polling results in the next run only if the polled directory has newer files that match the pattern specified. Default value: Selected | |||||||||||||||||||||||||||||||||||||||||
Number of retries | Specify the maximum number of retry attempts that the Snap must make in case there is a network failure, and the Snap is unable to read the target file. If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed. Ensure that the local drive has sufficient free disk space to store the temporary local file. Example: 3 Minimum value: 0 | |||||||||||||||||||||||||||||||||||||||||
Retry interval (seconds) | Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure. Minimum value: 1 Default value: 1 | |||||||||||||||||||||||||||||||||||||||||
Advanced properties | Use this field set to define specific settings for polling files. Click to add a new row for defining an advanced property. This field set comprises the following fields:
| |||||||||||||||||||||||||||||||||||||||||
SAS URI | Specify the URI of the Shared Access Storage (SAS) you need to access. Click to add the SAS URI. You can generate the SAS URI either from the SAS Generator Snap or from the Azure portal → Shared access signature. The supported SAS types are:
| |||||||||||||||||||||||||||||||||||||||||
Exit on first matches | Set this field to true to stop the Snap from executing after writing the first list of file paths that match the filter pattern, to the output view. If the field is not configured or is set to false, then the Snap continues to poll the directory until the Polling timeout is reached. Default value: false | |||||||||||||||||||||||||||||||||||||||||
|
|
Troubleshooting
| The library that we use for SFTP connections no longer supports deprecated signature protocols by default. (This changed with the 4.33 GA release.) | Add the algorithm to the serverProposal in the global.properties file.
Learn more: Configuration Options |
| If you have set the Polling Timeout value to a few seconds, it results in the S3 request getting canceled. | Increase the value of Polling Timeout (in seconds) for the Snap to work successfully. We recommend that you set the Polling Timeout value to the default value of 30 minutes or more to fetch all the S3 data. |
Examples
Writing out the List of Files in a Specific Directory Using the File Poller
In this example, you test the File Poller Snap to see whether it lists out files from a specific directory. You then write the output to a file and run the File Poller Snap again to check whether the new file was created as expected. To ensure that the File Poller Snap doesn't pick up any existing files, you use an unusual extension for this exercise.
Download this pipeline.
Expand | ||
---|---|---|
| ||
File PollerYou add a File Poller Snap and configure it to poll a directory for all files with the extension ".JSON2": You then use a combination of JSON Formatter and File Writer Snaps to process the output and write it out to a file. You use the date.now() function to give the file a name, so every time you run the pipeline, a new file is created. You use the JSON Formatter Snap with the default settings. File WriterWhen used in production, the output from the File Poller Snap can be used to trigger specific tasks as needed. In this example, you write it to a file: As expected, the file contains no output, as there is no file in the target directory with the extension ".JSON2". You now add a second File Poller Snap and configure it exactly as you configured the first one. Once again, you add a JSON Formatter and File Writer Snap with the same settings as for the previous pair. But this time, the file created is not blank: It lists out the file that you created using the first three Snaps in the pipeline: This tells you that the File Poller Snap works as expected with the settings you have used. Download this pipeline. |
Polling a Directory Using a Trigger Task from ServiceNow
In this example, you call a Trigger Task from ServiceNow to poll a directory for files of a specific type.
Download this pipeline
Expand | ||
---|---|---|
| ||
To make this example work, you must perform the following tasks:
Creating the File Poller PipelineYou design a pipeline containing the following Snaps:
You will note that the File Poller has open input and output views. This is because it receives data from the Trigger Task associated with it and returns processed data back to the same Trigger Task. CSV ParserYou use the CSV Parser Snap with the default settings: MapperYou configure the Mapper Snap to receive the parsed CSV data and map the message in the CSV document to the $msg variable: File PollerYou configure the File Poller Snap to poll the /QA/Documentation/File Poller/ directory for all files that match the pattern contained in the $msg variable, which you use as a file filter parameter: Creating a Trigger Task for the File Poller PipelineYou save the pipeline, click the (Create Task) button, and configure the Trigger Task: You click Update to complete setting up the task; then navigate to the Manager to view the Trigger Task's properties: You copy the Cloud URL and authorization bearer token, and navigate to ServiceNow to set up the API call. Creating a REST call in ServiceNowCreate a REST Call in ServiceNow by appending the authorization token to the Cloud URL that you copied in the previous step:
For details on how to set up REST calls using ServiceNow, see ServiceNow documentation. While configuring the REST call, ensure that:
You click Test to check the REST call. For a successful execution, the pipeline returns a list of files whose extension matches the value in the Content field: Download this pipeline. |
Downloads
Multiexcerpt include macro | ||||
---|---|---|---|---|
|
...