S3 File Reader

Owned by Kalpana Malladi
Last updated: Feb 15, 2024 by Lakshmi Manda
10 min read

On this Page

Overview

The S3 File Reader Snap reads data from an S3 bucket. When you provide a Version ID, the Snap reads a specific version of an S3 file object.

Important:

We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps. Learn more: Migration from Binary Snaps to Amazon S3 Snaps.

Snap Type

The S3 File Reader Snap is a Read-type Snap that reads data from an S3 bucket.

Prerequisites

  • IAM Roles for Amazon EC2.

  • The IAM_CREDENTIAL_FOR_S3 feature is to access S3 files from Groundplex nodes hosted in the EC2 environment. No Access-key ID and Secret key in the AWS S3 account is needed.

  • The IAM credential stored in the EC2 metadata provides access rights to the S3 buckets. 

    • IAM role is supported only in the Groundplex nodes hosted in the EC2 environment.

    • The IAM Role stored on the EC2 instance requires List, Read, and Write permissions.

    • S3 account validation is not supported when you enable the IAM role property. 

Learn more about IAM Roles for Amazon EC2.

  1. Open Manager.

  2. Open the Snaplexes tab of the project that contains the EC2-based Groundplex.

  3. Click the Groundplex to open its Properties.

  4. Open the Node Properties tab.

  5. Click + to add a new row in the Global properties section.

  6. Enter jvm_options in Key and -DIAM_CREDENTIAL_FOR_S3=TRUE in Value.

  7. Restart the JCC (node).

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

The current Snap functionality supports AWS S3 Cloud Service and applies to the AWSGovCloud setup.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 0

  • Max: 1

  • Mapper

  • File Writer

An upstream Snap is optional and any Snap with a document output view can be connected upstream (such as Mapper, File Writer, and so on). Any document with key-value pairs to evaluate expression properties in the S3 File Reader Snap. Each input document, if any, will cause one read operation of the Snap.

Output

Binary

 

  • Min: 0

  • Max: 1

  • CSV Parser

  • JASON Parser

  • XML Parser

Any Snap with a binary input view can be connected downstream, such as CSV Parser, JSON Parser, XML Parser, and so on. Binary data read from AWS S3 specified in the File property with header information about the binary stream. The binary data and header information can be previewed at the output of the Snap.

{ "content-length": "96258" "last-modified": { "_snaptype_datetime": "2014-06-26T23:27:01.000 UTC"} "content-disposition": "attachment; filename="leads.csv"" "content-location": "s3:///mr_test/leads.csv" "content-type": "text/csv" "etag": "730145bec198288e9f428193fde851b7" }

 

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( ): Indicates that you can add fields in the fieldset.

  • Remove icon ( ): Indicates that you can remove fields from the fieldset.

  • Upload icon ( ): Indicates that you can upload files.

Field Name

Field Type

 

Description

Field Name

Field Type

 

Description

Label*

 

Default Value: S3 File Reader
Example: S3 File Reader

String

Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

 

File

 

Default Value: s3:///
Example:

String/Expression/Suggestion

Specify the URL for the S3 file, from where the binary data is to be read. It must start with "s3:///". The suggest feature can be used to view the list of buckets, subdirectories and files. Bucket names are suggested if the property is empty or "s3:///". Once a bucket is selected, it can list subdirectories and files immediately below the bucket. Names of subdirectories end with a forward slash ("/"). The suggest feature is not supported if the propertie