Input 

Document

• Min: 0

• Max: 1

• Mapper

• Filter

• JSON Generator

An optional document to evaluate expressions in the Directory and/or File filter properties. Note that each input document will trigger the execution of the Snap.

Output

Document

• Min: 0

• Max: 1

• Mapper

• S3 Download

• JSON Formatter

A full path in each document as a value for a key "path". If multiple files match the filter, the same number of documents will be provided in the output view after each interval.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

• Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

• Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

• Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Field Name

Field Type

Description

Label*

Default Value: S3 Poller
Example: S3 Poller

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Bucket*

Default Value: N/A
Example: test-bucket

String/Expression/Sugestion

Specify or select an S3 bucket.

Directory 

Default Value:  N/A
Example: faculty/department/

String/Expression/Sugestion

Enter a directory as a prefix of an S3 object key ending with ‘/'. It should not start with '/’. Leave it empty if you are polling the bucket root.

Include subdirectories

Default state: Deselected

Checkbox/Expression

Select this checkbox to list all matching files in the specified directory and all subdirectories.

Filter (glob)

Default Value: N/A
Example:

• *.txt

• ab????xx.csv

String/Expression

Specify a GLOB pattern filter to match the file names in the directory. The Snap applies this filter to file names only, without including the directory path.

Polling interval (seconds)*

Default value: 30
Example: 10

Integer/Expression

Specify the time gap between each poll request (in seconds).

Polling timeout*

Default value: 30
Example: 20

Integer/Expression

Specify the maximum time for which the Snap continues to poll. If the Polling timeout is set to:

• Greater than 0, for example, 60 seconds, the polling stops after 60 seconds.

• 0, the Snap processes only one poll.

Polling-timeout unit

Default value: MINUTES
Example: SECONDS

Dropdown list

Specify a time unit for polling timeout.

Advanced properties

Maximum retries

Default value: 3
Example: 5

Integer/Expression

Enter the maximum number of retries the Snap performs when it fails to list a directory.

Only output on change

Default value: Selected

Checkbox/Expression

Select this checkbox, to output documents for matching files only when it starts up or when the file changes. Deselect this checkbox to output documents for matching files in every interval.

Exit on first matches

Default state: Deselected

Checkbox/Expression

Select this checkbox to enable the Snap to exit after the interval when one or more matches are found for the first time.

Snap execution

Default Value: Validate & Execute
Example: Execute only

Dropdown list

Select one of the following three modes in which the Snap executes:

• Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

• Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

• Disabled: Disables the Snap and all Snaps that are downstream from it.