Label*

Default Value: S3 File Reader
Example: S3 File Reader

String

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

File

Default Value: s3:///
Example:

• s3:///mybucket@s3.eu-west-1.amazonaws.com/test.json

• s3:///rds-sql-test-staging@s3.us-gov-east-1.amazonaws.com/test.csv

• _filename (A key/value pair with "filename" key should be defined as a pipeline parameter)

• $filename (A key/value pair with "filename" key should be defined the input document)

• VPC:

  • s3:///snaplogic-bucket@bucket.vpce-0e340d8d5b0eae90a-k9wbom46-us-west-2a.s3.us-west2.vpce.amazonaws.com

String/Expression/Suggestion

Specifythe URL for the S3 file, from where the binary data is to be read. It must start with "s3:///". The suggest feature can be used to view the list of buckets, subdirectories and files. Bucket names are suggested if the property is empty or "s3:///". Once a bucket is selected, it can list subdirectories and files immediately below the bucket. Names of subdirectories end with a forward slash ("/"). The suggest feature is not supported if the properties in the S3 Dynamic account are parameters.

This Snap supports S3 Virtual Private Cloud (VPC) endpoint.

Using Expressions:

Click the Expression Enabler to enable the expressions.

For example, if the File property is "s3:///mybucket/out_" + Date.now() + ".csv" then the evaluated filename is s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv.

Syntax:

s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

For region names and their details, see AWS Regions and Endpoints.

Version ID

Examples:   xvcnB8gPi37l3hbOzlsRFxjVwQ.numQz

Default value:  [None]

Expression/Suggestion

Enter or select S3 file version ID. If the property is empty, the Snap reads the latest version. The suggest feature can be used to view the list of version IDs for the S3 file in the File property. The suggest feature is not supported if the properties in the S3 Dynamic account are parameterized. Each line in the suggested list also includes the last modified date and the file size to help select a version. When the property value is entered manually, only the version ID is required. The Snap ignores the last modified date and size information of a version when it reads the file. If the versioning of a S3 bucket is not enabled, no version ID is suggested. The versions of the following cases will be omitted in the suggested list since their files cannot be downloaded: 

• If a file had existed before the versioning was enabled, its version does not have any version ID assigned to it.

• Version ID's with 'Deleted Marker' resource type are also omitted in the suggested list.

Version ID suggestion interval

Use this field set to read a specific version of S3 file object.

Field Set

Enter the time interval for the Version ID suggestion. Enter two rows to provide a start date and an end dates. If only one row is provided, the interval will be from the date until now. If empty, all version IDs will be suggested. This property may be useful when a given S3 file has many versions. This property is used for the Version ID suggestion only, and not used during the Snap preview or execution.

Year

Example:  2017

Default value:  [None]

Integer

Enter the year as a 4-digit integer.

Month

Examples:   9, 09, 12, and so on.

Default value:  [None]

Integer

Enter the month as an integer.

Date

Examples:   28, 09, 12, and so on.

Default value:  [None]

Integer

Enter the day of the month.

Zone

Examples:   US/Pacific

Default value:  [None]

Suggestion

Enter or select a time zone ID from the suggested list. May be empty for UTC. Please note only zone IDs in the suggested list are supported.

Enable staging

Default value: Not selected

Checkbox

If selected, the Snap downloads the source file into a local temporary file. When the download is completed, it streams the data from the temporary file to the output view. This property prevents the Snap from being blocked by slow downstream pipeline. The local disk should have sufficient free space as large as the expected file size. 

Number of retries

Example:  3

Minimum value: 0

Default value: 0

Integer/Expression

Specify the maximum number of retry attempts that the Snap must make in case there is a network failure, and the Snap is unable to read the target file.

If the value is larger than 0, the Snap overrides the Enable staging value to true and downloads the S3 file to a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap starts to stream the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.

Retry interval (seconds)

Example:  3

Minimum value: 1

Default value: 1

Integer/Expression

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Get Object Tags

Default value: Not selected

Checkbox

Select this checkbox to include object tags in the header of the output binary data. See Object Tagging for more information on object tags.

You must have the S3:GetObjectTagging permission to be able to use this feature.

Snap Execution

Default Value: 
Example: Validate & Execute

Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

• Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

• Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

• Disabled: Disables the Snap and all Snaps that are downstream from it.