In this article
Table of Contents | ||||
---|---|---|---|---|
|
Overview
S3 File Writer is a Write-type Snap that reads a binary data stream from its input view and writes it to an S3 file destination. If you provide values for File permissions, the Snap sets the permissions to the file.
Info |
---|
|
Prerequisites
None.
Supported Features
Works in Ultra Task Pipelines.
Limitations and Known Issues
None.
Snap Views
View Type | View Format | Number of Views | Examples of Upstream and Downstream Snaps | Description | ||
---|---|---|---|---|---|---|
Input | Document |
|
| Any binary data stream. | ||
Output | Document |
|
| If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:
| ||
Error | Document |
| N/A | The error view contains error, reason, resolution and stack trace. For more information, see Handling Errors with an Error Pipeline |
Optional Configuration
IAM Roles for Amazon EC2
To access S3 files from Groundplex nodes hosted in the EC2 environment without specifying Access-key ID and Secret key in AWS S3 account configured for the Snap, enable the ‘IAM_CREDENTIAL_FOR_S3’ feature. When you enable this feature, the IAM credential stored in EC2 metadata is used to access S3 buckets.
To enable the IAM_CREDENTIAL_FOR_S3 feature:
- Open Manager.
- Open the Snaplexes tab of the project that contains the EC2-based Groundplex.
- Click the Groundplex to open its Properties.
- Open the Node Properties tab.
- Add a new row in the Global properties section.
Specify jvm_options as the Key and -DIAM_CREDENTIAL_FOR_S3=TRUE as the Value.
Restart the JCC (node) to apply the changes. For more information about IAM Roles, refer to IAM Roles for Amazon EC2.
Behavior Change
Info | ||||
---|---|---|---|---|
| ||||
In the 4.23 release, the Snap used to write an S3 file and then set ACL if the ACL property is not empty. If the file write is successful and the ACL setting fails, both output document in the output view and error document in the error view were displayed. This has been fixed in 4.24 release by enabling the Snap to configure ACL when writing the file. Therefore, the Snap produces an error document in the error view if the ACL configuration fails, irrespective of the file write action passing or failing. |
Snap Settings
Info |
---|
Field names marked with an asterisk ( * ) are mandatory. |
Field Name | Field Type | Description | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Label* | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline. Default Value: S3 File Writer | |||||||||||||
File name* | String/Expression/Suggestion | Specify the URL for the S3 file, from where the binary data is to be read.
Prerequisite: The provided account must have Read access to the specified S3 bucket in order to read the file successfully. Using Expressions: This property can be an expression with the Expression enabler enabled. For example, if the File property is "s3:///mybucket/out_" + Date.now() + ".csv" then the evaluated filename is s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv. Syntax:
For region names and their details, see AWS Regions and Endpoints.
Default Value: s3:/// Examples:
| |||||||||||||
Suggest fully-qualified file names | Checkbox | Select this checkbox to include the region and authority of the S3 bucket in the associated paths that appear in the Suggestion list.
Default Value: Not Selected | |||||||||||||
File action* | Dropdown list | Specify the action to perform if the file already exists. The available options are:
Default Value: OVERWRITE | |||||||||||||
Write empty file | Checkbox | Select this checkbox to write an empty file when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no file is written regardless of the value of the property. Default Value: Not selected | |||||||||||||
Write header file | Checkbox | Select this checkbox to write a header file whose name is generated by appending ".header" to the value of the File name property. The header file provides metadata about the file such as content disposition or content type. The same header information is also included in the output view data, as shown in the "Expected output" section above, under the key "original". The binary data stream in the input view may contain header information about the binary data in the form of a document with key-value pair data.
Default Value: Not selected | |||||||||||||
Validate after write | Checkbox | Select this checkbox to allow the Snap to check if the file exists after writing the file. Default Value: Not selected | |||||||||||||
Number of retries | Integer/Expression | Specify the maximum number of retry attempts the Snap must make when it fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file.
Minimum value: 0 Default Value: 0 | |||||||||||||
Retry interval (seconds) | Integer/Expression | Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure. Minimum value: 1 Default Value: 1 | |||||||||||||
Buffer size(MB) | Integer | Specify the data (in MB) to load into the S3 bucket, at a time. Default Value: 10 MB
Refer to Upload Part for more information on uploading to S3. | |||||||||||||
Maximum upload threads | Integer | Specify the maximum number of threads to be used for the concurrent multipart upload. The minimum value allowed is 1. Default Value: 10
| |||||||||||||
AWS Canned ACL | Dropdown list | The predefined ACL grants (from AWS) to use when writing a file to S3. Choose a Canned ACL from the available options:
Learn more about AWS Canned ACLs. Default Value: Private | |||||||||||||
Access Control List | Use this field set to define the Access Control List (ACL) to the specified S3 file. This filed set contains the following fields:
| ||||||||||||||
Grantee | String/Expression/Suggestion | Select a grantee from the suggested list or enter a valid email address or a canonical ID associated to an AWS account. Canonical AWS ID can be obtained in the Security Credentials page of the AWS console.
Examples:
| |||||||||||||
Read | Checkbox | Grants permission to read the file. | |||||||||||||
View permissions | Checkbox | Grants permission to read the ACL. | |||||||||||||
Full control | Checkbox | Grants full control to the file. | |||||||||||||
User-defined object metadata | Use this field set to define key-value pairs for user-defined object metadata of an S3 object. For more information about user-defined object metadata, see Using Metadata This field set contains the following fields:
| ||||||||||||||
Key | String/Expression | Specify the key name of the object metadata.
| |||||||||||||
Value | String/Expression | Specify the value for the key entered above. | |||||||||||||
Object tags | Use this field set to define key-value pairs for object tags of an S3 object. Object tags enable you to categorize existing and new objects using key-value combinations. For details about the object tags, see Object Tagging. This field set contains the following fields:
| ||||||||||||||
Key | String/Expression | Specify the key name of the object tag.
| |||||||||||||
Value | String/Expression | Specify the value for the key entered above. | |||||||||||||
Snap Execution | Dropdown list |
|
Examples
Providing User-defined Object Metadata and Object Tags using the S3 File Writer Snap
This example is a basic use case for the S3 File Writer Snap. It also demonstrates how you can configure the Snap with custom object metadata and object tags to classify the data.
In the sample Pipeline, the S3 File Writer Snap is configured as follows with the User-defined object metadata and Object tags:
The following is a preview of the output data from the S3 File Writer Snap:
When the S3 File Reader Snap reads this data, it picks up the user-defined object metadata and object tags we defined, as show below:
Typical Configuration
Key configuration of the Snap lies in how the values are passed. Values can be passed to the Snap:
Without Expressions
For example, the File name is passed directly to the Snap in the image below.
With Expressions
The File name is passed using Pipeline Parameters:
Downloads
Multiexcerpt include macro | ||||
---|---|---|---|---|
|
Attachments | ||||||
---|---|---|---|---|---|---|
|
See Also
Insert excerpt | ||||||
---|---|---|---|---|---|---|
|