Versions Compared

Old Version 35

changes.mady.by.user Ankur Parekh

Saved on

compared with

New Version 36

changes.mady.by.user Chandna Mitra

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Modes

Snap type:

Read

Description:

This Snap polls the target directory and looks for file names matching the specified pattern. It continues polling at the intervals specified in the Polling interval property until the timeout (specified in the Polling timeout property) is reached. Once polling is done, the Snap lists all files whose names match the specified pattern.

Note

This Snap polls the target directory only; subdirectories, if any, are ignored. Use the Directory Browser Snap if you want to poll files in the directory and all subdirectories, as well as to poll a directory only once.


The File Poller Snap can be used in situations where an operation must be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling interval value is reached.

  • Expected upstream Snaps: Any Snap with a document output view, such as Mapper, JSON Generator.
  • Expected downstream Snaps: Any Snap with a document input view, such as File Reader, Mapper, JSON Formatter.
  • Expected input: An optional document to evaluate expressions in the Directory and/or File filter properties. Note that each input document will trigger the execution of the Snap.
  • Expected output: A full path in each document as a value for a key "path". If there are multiple files matching the filter, the same number of documents will be provided in the output view after each interval.


Code Block
[
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test1.csv"
        },
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test2.csv"
        }
]

Prerequisites:

Multiexcerpt include macro
nameEC2Prerequisite
pageFile Reader

Support and limitations:Ultra pipelines: 
  • Spark mode: Not supported in Spark mode.
    • Limitation
    • For S3 folders, the Snap currently supports polling the target directory for a maximum of 10,000 files. If there are more than that, the Snap does not provide any output.
    Account: 

    This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Binary Account for information on setting up these types of accounts. Account types supported by each protocol are as follows:

    ProtocolAccount types
    sldbno account
    s3AWS S3
    ftpBasic Auth
    sftpBasic Auth, SSH Auth 
    ftpsBasic Auth
    hdfsno account
    webhdfsno account
    smbSMB
    wasbAzure Storage
    wasbsAzure Storage
    gs

    Google Storage

    fileLocal file system


    Note

    The FTPS file protocol works only in explicit mode. The implicit mode is not supported.


    Required settings for account types are as follows:

    Account TypeSettings
    Basic AuthUsername, Password
    AWS S3Access-key ID, Secret key
    SSH AuthUsername, Private key, Key Passphrase
    SMBDomain, Username, Password
    Azure StorageAccount name, Primary access key
    Google StorageApproval prompt, Application scope, Auto-refresh token
    (Read-only properties are Access token, Refresh token, Access token expiration, OAuth2 Endpoint, OAuth2 token and Access type.)


    Views:


    InputThis Snap has at most one document input view.
    OutputThis Snap has exactly one document output view.
    ErrorThis Snap has at most one document error view.


    Settings

    Label

    		Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

    Directory 


    This property is a URL path to the directory where files will be searched. The expected syntax is: 

      [protocol]://[host][:port]/[path]

    The supported file protocols are:

    • s3:
    • file:

    • ftp:

    • ftps:
    • sftp: 
    • hdfs:
    • webhdfs:

    • sldb: 

    • smb:

    • wasb:

    • wasbs:

    • gs:

    Example: 

      • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

        For region names and their details, see AWS Regions and Endpoints.

      • sftp://ftp.snaplogic.com:22/home/test/dir

      • ftp://ftp.snaplogic.com/test/csv

      • $directory 

      • _directory (A key-value pair with "directory" key should be defined as a pipeline parameter. Ensure that the '=' button is enabled when using parameters.)

      • file:///D:/testFolder/  (if the Snap is executed in the Windows Groundplex and needs to access the D: drive)

      • wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/

      • gs:///testBucket/testDir/ 

    Default value:  [None]


    Note
    • The protocol and the rest of the URL should be separated by "://". The host name and the port number should be between "://" and "/".
    • Not all file protocols support "//", use "///" instead. For example, if polling files in SLDB and S3 (see the examples shown above).  
    • This property should be an absolute path for all protocols except SLDB. For SLDB, the Snap can access only the same project directory or the shared project directory, and cannot access other project directories.
    • If you want this property to refer to the SLDB project (or shared project) directory where the pipeline of this Snap belongs to, enter "sldb:///" or leave it blank.
    • If the pipeline is created in a project other than the shared project and you want this property to refer to the shared project, enter "shared" or "sldb:///shared".
    • If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the SLDB and S3 protocols.


    File filter

    Required. A GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document.
    Example:

    • *.txt
    • ab????xx.csv

    Default value:  [None]

    Excerpt


    Expand
    titleGlob Pattern Interpretation Rules

    Use glob patterns in this filter to select one or more files in the directory. For example:

    • *.java Matches file names ending in .java.
    • *.* Matches file names containing a dot.
    • *.{java,class} Matches file names ending with .java or .class.
    • foo.? Matches file names starting with foo. and a single character extension.

    The following rules are used to interpret glob patterns:

    • The * character matches zero or more characters of a name component without crossing directory boundaries.
    • The ? character matches exactly one character of a name component.
    • The backslash character (\) is used to escape characters that would otherwise be interpreted as special characters. For example, the expression \\ matches a single backslash, and "\{" matches a left brace.
    • The ! character is used to exclude matching files from the output. 
    • The [ ] characters are a bracket expression that match a single character of a name component out of a set of characters. For example, [abc] matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range; so, [a-z] specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed; so, [abce-g] matches 'a'", 'b', 'c', 'e', 'f' or 'g'. If the character after the '[' is an '!', then it is used for negation; so, [!a-c] matches any character except 'a', 'b', or 'c'.
    • Within a bracket expression, the *, ?, and \ characters match themselves. The (-) character matches itself if it is the first character within the brackets, or the first character after the '!', if negating.
    • The { } characters are a group of subpatterns, where the group matches if any subpattern in the group matches. The ',' character is used to separate subpatterns. Groups cannot be nested.
    • Leading period / dot characters in file names are treated as regular characters in match operations. For example, the '*' glob pattern matches file name '.login'.
    • Some special characters are not supported. A partial list of unsupported special characters: #, ^, â, ê, î, ç, ¿, SPACE.



    Polling interval in seconds

    Required. The time-gap between each poll request (in seconds).

    Example: 10

    Default value: 30

    Polling timeout

    Required. A period of time after which file polling must end. You specify the number here and the time unit in the next property. Its unit is selected in the next property.

    Example:

    • -1 (to poll indefinitely)
    • 0 (to poll once)
    • 60 (its unit shown in the next property)

    Default value: 30

    Note

    Configure this property based on the expected number of files in the target directory. If there are many files and this property's value is small, the Snap may complete the operation and stop before the file is found. 


    Polling-timeout unit

    Unit for the polling timeout. Allowed values are SECONDS, MINUTES and HOURS.

    Example: SECONDS

    Default value: MINUTES

    Only Output on Change

    Select this check box to instruct the Snap to provide an output only when there is a change in the contents of the polled directory. When selected, the Snap provides an output during its initial run if it finds matching documents. However, it provides polling results in the next run only if the polled directory has newer files that match the pattern specified. 

    Default value: Selected

    Number of retries

    Specifies the maximum number of retry attempts in case of a network failure.

    Example:  3

    Minimum value: 0

    Default value: 0

    Multiexcerpt include macro
    nameretries
    pageFile Reader

    Retry interval (seconds)

    Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

    Example:  3

    Minimum value: 1

    Default value: 1

    Multiexcerpt include macro
    nameSASURI_Name
    pageFile Reader
    Multiexcerpt include macro
    nameSASURI_Description_NoAccountSAS
    pageDirectory Browser

    Multiexcerpt include macro
    nameSnap Execution
    pageAnaplan Read


    Multiexcerpt include macro
    nameSnap_Execution_Introduced
    pageAnaplan Read

    Examples

    Writing out the List of Files in a Specific Directory Using the File Poller

    In this example, you test the File Poller Snap to see whether it reliably lists out files from a specific directory. You then write the output to a file and run the File Poller Snap again to check whether the new file was created as expected. To ensure that the File Poller Snap doesn't pick up any existing files, you use an unusual extension for this exercise.

    Download this pipeline.

    Expand
    titleUnderstanding the pipeline

    File Poller

    You add a File Poller Snap and configure it to poll a directory for all files with the extension ".JSON2":

    You then use a combination of JSON Formatter and File Writer Snaps to process the output and write it out to a file. You use the date.now() function to give the file a name, so every time you run the pipeline, a new file is created.

    You use the JSON Formatter Snap with the default settings.

    File Writer

    When used in production, the output from the File Poller Snap can be used to trigger specific tasks as needed. In this example, you write it to a file:

    As expected, the file contains no output, as there is no file in the target directory with the extension ".JSON2".

    You now add a second File Poller Snap and configure it exactly as you configured the first one. Once again, you add a JSON Formatter and File Writer Snap with the same settings as for the previous pair. But this time, the file created is not blank: It lists out the file that you created using the first three Snaps in the pipeline:

    This tells you that the File Poller Snap works as expected with the settings you have used.

    Download this pipeline.


    Polling a Directory Using a Trigger Task from ServiceNow

    In this example, you call a Trigger Task from ServiceNow to poll a directory for files of a specific type. 

    Download this pipeline

    Expand
    titleUnderstanding the pipeline

    To make this example work, you must perform the following tasks:

    1. Create a pipeline that receives specifications from ServiceNow on the file type for which you want to poll a directory.
    2. Create a Trigger Task that will make the pipeline available as a REST API.
    3. Create a REST call in ServiceNow that will trigger the Task created in Step 2 above.

    Creating the File Poller Pipeline

    You design a pipeline containing the following Snaps:

    • CSV Parser: Parses the data coming in from ServiceNow as a CSV document.
    • Mapper: Makes the contents of the input CSV document available as a parameter.
    • File Poller: Polls a specific directory for files matching the pattern specified in the parameter received from the Mapper Snap.

    You will note that the File Poller has open input and output views. This is because it receives data from the Trigger Task associated with it and returns processed data back to the same Trigger Task.

    CSV Parser

    You use the CSV Parser Snap with the default settings:

    Mapper

    You configure the Mapper Snap to receive the parsed CSV data and map the message in the CSV document to the $msg variable:

    File Poller

    You configure the File Poller Snap to poll the /QA/Documentation/File Poller/ directory for all files that match the pattern contained in the $msg variable, which you use as a file filter parameter:

    Creating a Trigger Task for the File Poller Pipeline

    You save the pipeline, click the  (Create Task) button, and configure the Trigger Task:

    You click Update to complete setting up the task; then navigate to the Manager to view the Trigger Task's properties:

    You copy the Cloud URL and authorization bearer token, and navigate to ServiceNow to set up the API call.

    Creating a REST call in ServiceNow

    Create a REST Call in ServiceNow by appending the authorization token to the Cloud URL that you copied in the previous step:

    Paste code macro
    <Cloud_URL>?bearer_token=<Bearer_Token>

    For details on how to set up REST calls using ServiceNow, see ServiceNow documentation.

    While configuring the REST call, ensure that:

    • HTTP Method is POST.
    • Endpoint contains the Cloud URL concatenated with the bearer token.
    • HTTP Parameters > Content contains the variable and extension value that you want to use.

    You click Test to check the REST call. For a successful execution, the pipeline returns a list of files whose extension matches the value in the Content field:


    Download this pipeline.


    Attachments
    previewfalse
    uploadfalse
    oldfalse
    patterns*.slp

    Insert excerpt
    Binary Snap Pack
    Binary Snap Pack
    nopaneltrue