ETL Transformations & Data Flow

This Snap extracts data from S3 buckets. 

Input & Output

• Expected input: An upstream Snap is optional and any Snap with a document output view can be connected upstream (such as Mapper, File Writer, and so on). Any document with key-value pairs to evaluate expression properties in the S3 File Reader Snap. Each input document, if any, will cause one read operation of the Snap.
• Expected output: Any Snap with a binary input view can be connected downstream, such as CSV Parser, JSON Parser, XML Parser, and so on. Binary data read from AWS S3 specified in the File property with header information about the binary stream. The binary data and header information can be previewed at the output of the Snap. An example is: 

{
	"content-length": "96258"
	"last-modified":  {
		"_snaptype_datetime": "2014-06-26T23:27:01.000 UTC"}
	"content-disposition": "attachment; filename="leads.csv""
	"content-location": "s3:///mr_test/leads.csv"
	"content-type": "text/csv"
	"etag": "730145bec198288e9f428193fde851b7"
}

Works in Ultra Pipelines.

Account & Access

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.

Required settings for account types are as follows: 

• AWS S3 - Access-key ID, Secret key, Security token.
• S3 Dynamic - Access-key ID, Secret key, Security token, Server-side encryption.

Views

Required. This property specifies the URL for the S3 file, from where the binary data is to be read. It must start with "s3:///". The suggest feature can be used to view the list of buckets, subdirectories and files. Bucket names are suggested if the property is empty or "s3:///". Once a bucket is selected, it can list subdirectories and files immediately below the bucket. Names of subdirectories end with a forward slash ("/"). The suggest feature is not supported if the properties in the S3 Dynamic account are parameters.

Using Expressions:

This property can be an expression with the "=" button pressed.

For example, if the File property is "s3:///mybucket/out_" + Date.now() + ".csv" then the evaluated filename is s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv.

Syntax:

s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

For region names and their details, see AWS Regions and Endpoints.

Examples: 

• s3:///mybucket@s3.eu-west-1.amazonaws.com/test.json
• s3:///rds-sql-test-staging@s3.us-gov-east-1.amazonaws.com/test.csv
• _filename (A key/value pair with "filename" key should be defined as a pipeline parameter)
• $filename (A key/value pair with "filename" key should be defined the input document)

Default value:  s3:///

Enter or select S3 file version ID. If the property is empty, the Snap reads the latest version. The suggest feature can be used to view the list of version IDs for the S3 file in the File property. The suggest feature is not supported if the properties in the S3 Dynamic account are parameterized. Each line in the suggested list also includes the last modified date and the file size to help select a version. When the property value is entered manually, only the version ID is needed. The Snap ignores the last modified date and size information of a version when it reads the file. If the versioning of a S3 bucket is not enabled, no version ID is suggested. The versions of the following cases will be omitted in the suggested list since their files cannot be downloaded: 

• If a file had existed before the versioning was enabled, its version does not have any version ID assigned to it.
• Version ID's with 'Deleted Marker' resource type are also omitted in the suggested list.

Examples:   xvcnB8gPi37l3hbOzlsRFxjVwQ.numQz

Default value:  [None]

Enter the year as a 4-digit integer.

Example:  2017

Default value:  [None]

Enter the month as an integer.

Examples:   9, 09, 12, and so on.

Default value:  [None]

Enter the day of the month.

Examples:   28, 09, 12, and so on.

Default value:  [None]

Enter or select a time zone ID from the suggested list. May be empty for UTC. Please note only zone IDs in the suggested list are supported.

Examples:   US/Pacific

Default value:  [None]

If selected, the Snap downloads the source file into a local temporary file. When the download is completed, it streams the data from the temporary file to the output view. This property prevents the Snap from being blocked by slow downstream pipeline. The local disk should have sufficient free space as large as the expected file size. 

Default value: Not selected

Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Example:  3

Minimum value: 1

Default value: 1

Select this check box to include object tags in the header of the output binary data. See Object Tagging for more information on object tags.

You must have the S3:GetObjectTagging permission to be able to use this feature.

Default value: Not selected