Versions Compared

Old Version 71

changes.mady.by.user Kalpana Malladi

Saved on

compared with

New Version Current

changes.mady.by.user Kalpana Malladi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

...

Snap type:

...

Read

...

Description:

This Snap polls the target directory and looks for file names matching the specified pattern. It continues polling at the intervals specified in the Polling interval property until the timeout (specified in the Polling timeout property) is reached. Once polling is done, the Snap lists all files whose names match the specified pattern.

Note

This Snap polls the target directory only; subdirectories, if any, are ignored. Use the Directory Browser Snap if you want to poll files in the directory and all subdirectories, as well as to poll a directory only once.

The File Poller Snap can be used in situations where an operation must be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling interval value is reached.

  • Expected upstream Snaps: Any Snap with a document output view, such as Mapper, JSON Generator.
  • Expected downstream Snaps: Any Snap with a document input view, such as File Reader, Mapper, JSON Formatter.
  • Expected input: An optional document to evaluate expressions in the Directory and/or File filter properties. Note that each input document will trigger the execution of the Snap.
  • Expected output: A full path in each document as a value for a key "path". If there are multiple files matching the filter, the same number of documents will be provided in the output view after each interval.
Code Block
[
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test1.csv"
        },
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test2.csv"
        }
]

...

Multiexcerpt include macro
nameEC2Prerequisite
pageFile Reader

Multiexcerpt include macro
nameFTPS_Prerequisite
templateDataeJyLjgUAARUAuQ==
pageDirectory Browser
addpanelfalse

...

The Snap is expected to fail if there is no account selected. However, the Snap may execute successfully without any account if all the following conditions exist:

  • The Snap is executed in an EC2-instance Snaplex where your pipeline runs with an IAM role.

  • The S3 bucket accessed by the Snap includes the necessary permissions for use with the specific IAM role.

  • The following global property is set as a node property in the plex:
    jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

Image Removed

Multiexcerpt include macro
nameKI
templateData[]
pageZipFile Write
addpanelfalse

...

Behavior Change

...

The File Poller Snap now honors the value specified in the Polling timeout field instead of polling indefinitely in case of poor file polling operations. To handle indefinite polling operations the polling is done in a separate thread. However, when the execution time exceeds the value specified in the Polling timeout, a timeout exception is written to the log to prevent the polling from getting stuck and the Snap continues polling depending on the Polling timeout.

  • If the Polling timeout value is greater than 0, the Snap polls until the end of polling window.

  • If it is less than 0, the Snap stops polling.

  • If it is -1, the Snap continues polling.

...

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.

Account types supported by each protocol are as follows:

ProtocolAccount types
sldbno account
s3AWS S3
ftpBasic Auth
sftpBasic Auth, SSH Auth 
ftpsBasic Auth
hdfsno account
smbSMB
wasbAzure Storage
wasbsAzure Storage
gs

Google Storage

fileLocal file system
Note

The FTPS file protocol works only in explicit mode. The implicit mode is not supported.

Required settings for account types are as follows:

Account TypeSettings
Basic AuthUsername, Password
AWS S3Access-key ID, Secret key
SSH AuthUsername, Private key, Key Passphrase
SMBDomain, Username, Password
Azure StorageAccount name, Primary access key
Google StorageApproval prompt, Application scope, Auto-refresh token
(Read-only properties are Access token, Refresh token, Access token expiration, OAuth2 Endpoint, OAuth2 token and Access type.)

...

InputThis Snap has at most one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

...

Settings

...

Label

...

In this article

Table of Contents
minLevel1
maxLevel3
outlinefalse
styledefault
typelist
printabletrue

Overview

You can use this Snap to poll the target directory and find file names matching the specified pattern.

Info

  • The Snap continues polling at the intervals specified in the Polling interval property until the timeout (specified in the Polling timeout property) is reached. After polling is done, the Snap lists all files whose names match the specified pattern.

  • This Snap can be used in situations where an operation must be triggered when a specific file is found in the target directory. The pipeline can be configured with additional Snaps to process the Snap's output and delete the matched file before the Polling interval value is reached.

  • The File Poller Snap uses the case-sensitive filter pattern, regardless of the operating system.

Note

  • This Snap polls the target directory only. Subdirectories, if any, are ignored. Use the Directory Browser Snap if you want to poll files in the directory and all subdirectories, and to poll a directory only once.

...

Snap Type

The File Poller Snap is a Read-type Snap.

Prerequisites

Multiexcerpt include macro
macro_uuid36fcee4c-2e9a-48ac-898f-956567780f54
nameEC2Prerequisite
pageFile Reader

Multiexcerpt include macro
nameFTPS_Prerequisite
templateDataeJyLjgUAARUAuQ==
pageDirectory Browser
addpanelfalse

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

For S3 folders, the Snap currently supports polling the target directory for a maximum of 10,000 files. If there are more than that, the Snap does not provide any output.

Known Issues

The Snap is expected to fail if there is no account selected. However, the Snap may execute successfully without any account if all the following conditions exist:

  • The Snap is executed in an EC2-instance Snaplex where your pipeline runs with an IAM role.

  • The S3 bucket accessed by the Snap includes the necessary permissions for use with the specific IAM role.

  • The following global property is set as a node property in the plex:
    jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

    Image Added

  • Multiexcerpt include macro
    nameKI
    templateData[]
    pageZipFile Write
    addpanelfalse

Behavior Change

The File Poller Snap now honors the value specified in the Polling timeout field instead of polling indefinitely in case of poor file polling operations. To handle indefinite polling operations the polling is done in a separate thread. However, when the execution time exceeds the value specified in the Polling timeout, a timeout exception is written to the log to prevent the polling from getting stuck and the Snap continues polling depending on the Polling timeout.

  • If the Polling timeout value is greater than 0, the Snap polls until the end of polling window.

  • If it is less than 0, the Snap stops polling.

  • If it is -1, the Snap continues polling.

Supported Protocols

Account types supported by each protocol are as follows:

Protocol

Account types

sldb

no account

s3

AWS S3

ftp

Basic Auth

sftp

Basic Auth, SSH Auth 

ftps

Basic Auth

hdfs

no account

smb

SMB

wasb

Azure Storage

wasbs

Azure Storage

gs

Google Storage

file

Local file system

Note

The FTPS file protocol works only in explicit mode. The implicit mode is not supported.

Required settings for account types are as follows:

Account Type

Settings

Basic Auth

Username, Password

AWS S3

Access-key ID, Secret key

SSH Auth

Username, Private key, Key Passphrase

SMB

Domain, Username, Password

Azure Storage

Account name, Primary access key

Google Storage

Approval prompt, Application scope, Auto-refresh token
(Read-only properties are Access token, Refresh token, Access token expiration, OAuth2 Endpoint, OAuth2 token and Access type.)

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

 

  • Min: 1

  • Max: 1

  • Mapper

  • JSON Generator

An optional document to evaluate expressions in the Directory and/or File filter properties. Note that each input document will trigger the execution of the Snap.

Output

Document

 

  • Min: 1

  • Max: 1

  • Mapper

  • File Reader

  • JSON Formatter

A full path in each document as a value for a key "path". If multiple files match the filter, the same number of documents will be provided in the output view after each interval.

Code Block
[
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test1.csv"
        },
        {
                "path" :  "sftp://sftp.smart.com/home/voo/test2.csv"
        }
]

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

Info

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the field set.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the field set.

  • Upload icon ((blue star) ): Indicates that you can upload files.

Field Name

Field Type

Description

Label*


Default Value: File Poller
Example: File Poller

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Directory 

This property is a URL path to the directory where files will be searched. The expected syntax is: 

  [protocol]://[host][:port]/[path]

The supported file protocols are:

  • s3:
  • file:

  • ftp:

  • ftps:
  • sftp: 
  • hdfs:

  • sldb: 

  • smb:

  • wasb:

  • wasbs:

  • gs:

Example: 

    • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

      For region names and their details, see AWS Regions and Endpoints.

    • sftp://ftp.snaplogic.com:22/home/test/dir

    • ftp://ftp.snaplogic.com/test/csv

    • $directory 

    • _directory (A key-value pair with "directory" key should be defined as a pipeline parameter. Ensure that the '=' button is enabled when using parameters.)

    • file:///D:/testFolder/  (if the Snap is executed in the Windows Groundplex and needs to access the D: drive)

    • wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/

    • gs:///testBucket/testDir/ 

...

Directory 

Default Value:  N/A
Example: 

  • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

    For region names and their details, see AWS Regions and Endpoints.

  • sftp://ftp.snaplogic.com:22/home/test/dir

  • ftp://ftp.snaplogic.com/test/csv

  • $directory 

  • _directory (A key-value pair with "directory" key should be defined as a pipeline parameter. Ensure that the '=' button is enabled when using parameters.)

  • file:///D:/testFolder/  (if the Snap is executed in the Windows Groundplex and needs to access the D: drive)

  • wasb:///Snaplogic/testDir/ or wasbs:///Snaplogic/testDir/

  • gs:///testBucket/testDir/ 

String/Expression

Specify the URL path to the directory where files will be searched in the following format: 

  [protocol]://[host][:port]/[path]

The supported file protocols are:

  • s3:

  • file:

  • ftp:

  • ftps:

  • sftp: 

  • hdfs:

  • sldb: 

  • smb:

  • wasb:

  • wasbs:

  • gs:

Note

  • The protocol and the rest of the URL should be separated by "://". The host name and the port number should be between "://" and "/".

  • Not all file protocols support "//", use "///" instead. For example, if polling files in SLDB and S3 (see the examples shown above).  

  • This property should be an absolute path for all protocols except SLDB. For SLDB, the Snap can access only the same project directory or the shared project directory, and cannot access other project directories.

  • If you want this property to refer to the SLDB project (or shared project) directory where the pipeline of this Snap belongs to, enter "sldb:///" or leave it blank.

  • If the pipeline is created in a project other than the shared project and you want this property to refer to the shared project, enter "shared" or "sldb:///shared".

  • If the port number is omitted, a default port for the protocol is used. The hostname and port number are omitted in the SLDB and S3 protocols.

  • Ensure the file name, folder name, or the file path does not contain '?' character because it is not fully supported and when present, the Snap might fail.

  • This Snap supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

File filter

...

*


Default Value: N/A
Example:

  • *.txt

  • ab????xx.csv

String/Expression

Specify a GLOB pattern to be applied to select one or more files in the directory. The File filter property can be a JavaScript expression which will be evaluated with values from the input view document

...

.

...

  [None]

Excerpt


Expand
titleGlob Pattern Interpretation Rules

Use glob patterns in this filter to select one or more files in the directory. For example:

  • *.java Matches file names ending in .java.
  • *.* Matches file names containing a dot.
  • *.{java,class} Matches file names ending with .java or .class.
  • foo.? Matches file names starting with foo. and a single character extension.

The following rules are used to interpret glob patterns:

  • The * character matches zero or more characters of a name component without crossing directory boundaries.
  • The ? character matches exactly one character of a name component.
  • The backslash character (\) is used to escape characters that would otherwise be interpreted as special characters. For example, the expression \\ matches a single backslash, and "\{" matches a left brace.
  • The ! character is used to exclude matching files from the output. 
  • The [ ] characters are a bracket expression that match a single character of a name component out of a set of characters. For example, [abc] matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range; so, [a-z] specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed; so, [abce-g] matches 'a'", 'b', 'c', 'e', 'f' or 'g'. If the character after the '[' is an '!', then it is used for negation; so, [!a-c] matches any character except 'a', 'b', or 'c'.
  • Within a bracket expression, the *, ?, and \ characters match themselves. The (-) character matches itself if it is the first character within the brackets, or the first character after the '!', if negating.
  • The { } characters are a group of subpatterns, where the group matches if any subpattern in the group matches. The ',' character is used to separate subpatterns. Groups cannot be nested.
  • Leading period / dot characters in file names are treated as regular characters in match operations. For example, the '*' glob pattern matches file name '.login'.
  • Some special characters are not supported. A partial list of unsupported special characters: #, ^, â, ê, î, ç, ¿, SPACE.


Polling interval in seconds

...

*


Default value: 30
Example: 10

Integer

Specify the time gap between each poll request (in seconds).

...

Polling timeout*

Default value: 30

...

Polling timeout

Example: 20

Integer

Specify a period of time after which file polling must end

...

.

...

Example:

If the Polling timeout is set to

...

:

  • Greater than

...

  • 0, for example, 60 seconds, the polling stops after 60 seconds.

...

  • 0, the Snap processes only one poll.

...

  • -1, the Snap polls continually.

Default value: 30

Note

Configure this field based on the expected number of files in the

...

target directory.

...

 If there

...

are many files and this field's

...

value is small,

...

 the Snap may complete the operation

...

 and stop before the file is found. 

Polling-timeout unit

...


Default value: MINUTES
Example: SECONDS

...

Dropdown list

Specify a value for polling timeout.

Only Output on Change

Default value: Selected

Checkbox

Select this check box to instruct the Snap to provide an output only when there is a change in the contents of the polled directory. When selected, the Snap provides an output during its initial run if it finds matching documents. However, it provides polling results in the next run only if the polled directory has newer files that match the pattern specified. 

Number of retries

Minimum value: 0
Default value

...

: 0
Example:  3

Integer

Specify the maximum number of retry attempts that the Snap must make in case there is a network failure, and the Snap is unable to read the target file.

If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.

Ensure that the local drive has sufficient free disk space to store the temporary local file.

...

Retry interval (seconds)

Minimum value:

...

1

Default value

...

: 1
Example:  3

Integer

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

...

Default value: 1
Example:  3


Advanced properties

Use this field set to define specific settings for polling files

...

.

...

Properties

...

Dropdown list

Choose either of the following

...

options:

  • SAS URI

  • Exit on first matches

Values

String/Expression

  • If you choose SAS URI

...

  • , specify the URI of the Shared Access Storage (SAS) you need to access

...

  • . You can generate the SAS URI either from the SAS Generator Snap or from the Azure portal → Shared access signature.

...

  • The supported SAS types are:

    • Service SAS on a container

    • Service SAS on blob

Note

  • Ensure that the URI is specified in the format described here.

  • If you provide SAS URI in this field, then:

    • the Primary access key given in the account settings is overridden

...

    • during authentication. If you do not provide the SAS URI, the Snap considers the Primary access key in the account settings.

    • only this URL is used and the Snap ignores the SAS URI settings that you have configured in the associated account.

  • If you choose Exit on first matches

...

  • , set this field to true to stop the Snap from executing after writing the first list of file paths that match the filter pattern

...

  • to the output view.

...

  •  If the field is not configured or is set to false, then the Snap continues to poll the directory until the Polling timeout is reached.

Snap execution

Default

...

Value: Validate & Execute
Example: Execute only

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute

Troubleshooting

Error

Reason

Resolution

Algorithm negotiation fail: algorithmName="server_host_key" jschProposal="<algorithms>" serverProposal="ssh-rsa"

The library that we use for SFTP connections no longer supports deprecated signature protocols by default.

...

This changed with the 4.33 GA release.

...

Add the algorithm to the serverProposal in the global

...

properties

...

.
You can also enable support for RSA-SHA1 authentication in the Node Properties tab on the Update Snaplex dialog in SnapLogic Manager.

  1. In the Node Properties tab of your target Snaplex, add the following

...

  1. Key Value pair under Global Properties:
    Key: jcc.jvm_options
    Value: -Djsch.server_host_key=ssh-rsa -Djsch.client_pubkey=ssh-rsa

  2. Click Update, and then restart the Snaplex node.

Learn more: Configuration Options

com.amazonaws.AbortedException - Cannot access AWS S3 service

If you have set the Polling Timeout value to a few seconds, it results in the S3 request getting canceled.

Increase the value of Polling Timeout (in seconds) for the Snap to work successfully.

We recommend that you set the Polling Timeout value to the default value of 30 minutes or more to fetch all the data from S3

...

.

Examples

...

Write a List of Files in a Specific

...

Directory

This example pipeline demonstrates how to list out files from a specific directory.

...

After the Poller Snap lists the files, then write the output to a file and run the File Poller Snap again to check whether the new file was created as expected. To ensure that the File Poller Snap doesn't pick up any existing files, you use an unusual extension for this

...

example.

...

Download this pipeline.

...

titleUnderstanding the pipeline

File Poller

...

Configure the File Poller Snap to poll a directory for all files with the extension ".JSON2"

...

.

Image Removed

...

Connect JSON Formatter and File Writer Snaps to process the output and write

...

it to a file. Use the JSON Formatter Snap with the default settings. In the File Writer Snap, use the date.now() function to give the file a name, so a new file is created every time you run the pipeline

...

.

You use the JSON Formatter Snap with the default settings.

File Writer

...

When used in production, the output from the File Poller Snap can be used to trigger specific tasks as needed. In this example, you write it to a file

...

Image Removed

. As expected, the file contains no output, as there is no file in the target directory with the extension ".JSON2".

...

...

Next, add

...

the second File Poller Snap and configure it exactly as

...

the first one. Once again, you add a JSON Formatter and File Writer Snap with the same settings as for the previous pair. But this time, the file created is not blank: It lists out the file that you created using

...

Image Removed

...

the first three Snaps in the pipeline:

...

Download this pipeline.

...

Poll a Directory Using a Trigger Task from ServiceNow

In this example, you call a Trigger Task from ServiceNow to poll a directory for files of a specific type. 

...

Download this pipeline

Expand
titleUnderstanding the pipeline

To make this example work, you must perform the following tasks:

  1. Create a pipeline that receives specifications from ServiceNow on the file type for which you want to poll a directory.

  2. Create a Trigger Task that will make the pipeline available as a REST API.

  3. Create a REST call in ServiceNow that will trigger the Task created in Step 2 above.

...

Create the File Poller Pipeline

You design a pipeline containing the following Snaps:

  • CSV Parser: Parses the data coming in from ServiceNow as a CSV document.

  • Mapper: Makes the contents of the input CSV document available as a parameter.

  • File Poller: Polls a specific directory for files matching the pattern specified in the parameter received from the Mapper Snap.

You will note that the File Poller has open input and output views. This is because it receives data from the Trigger Task associated with it and returns processed data back to the same Trigger Task.

CSV Parser

...

:Use the CSV Parser Snap with the default settings:

...

Image Added

Mapper

...

: Configure the Mapper Snap to receive the parsed CSV data and map the message in the CSV document to the $msg variable:

...

Image Added

File Poller

...

: Configure the File Poller Snap to poll the /QA/Documentation/File Poller/ directory for all files that match the pattern contained in the $msg variable, which you use as a file filter parameter:

...

Image Added

...

Create a Trigger Task for the File Poller Pipeline

...

: Save the pipeline, click the Image Modified (Create Task) button, and configure the Trigger Task:

...

Image Added

...

Click Update to complete setting up the task; then navigate to the Manager to view the Trigger Task's properties:

...

Image Added

You copy the Cloud URL and authorization bearer token, and navigate to ServiceNow to set up the API call.

...

Create a REST call in ServiceNow

...

:Create a REST Call in ServiceNow by appending the authorization token to the Cloud URL that you copied in the previous step:

Paste code macro
<Cloud_URL>?bearer_token=<Bearer_Token>

...

Image Added

For details on how to set up REST calls using ServiceNow, see ServiceNow documentation.

While configuring the REST call, ensure that:

  • HTTP Method is POST.

  • Endpoint contains the Cloud URL concatenated with the bearer token.

  • HTTP Parameters > Content contains the variable and extension value that you want to use.

You click Test to check the REST call. For a successful execution, the pipeline returns a list of files whose extension matches the value in the Content field:

...

Image Added

Download this pipeline.

Downloads

Multiexcerpt include macro
namedownload_instructions
pageOpenAPI

Attachments
previewfalse
uploadfalse
oldfalse
patterns*.slp

...

Snap Pack History

Insert excerpt
Binary Snap Pack
Binary Snap Pack
nameBinary_SPH
nopaneltrue