In this article
...
You can use this Snap to list the attributes of S3 objects in a given bucket matching the prefix. The attributes provided in the output document are bucket, objectKey, path, size, lastModified and owner. It internally lists all S3 objects with a given bucket name and prefix, and then applies the filter if any. Note that AWS S3 does not define any folder object and the '/' characters are part of the S3 object key names.
<overview image>
Snap Type
<Snap name> Snap is a READ/WRITE/PARSE/TRANSFORM/FLOW/FORMAT-type Snap that reads/fetches/writes/parses/executes/transforms/calls/creates…
...
Type | Format | Number of Views | Examples of Upstream and Downstream Snaps | Description | |
---|---|---|---|---|---|
Input | Document |
|
| Requires the EDI data and internal ID as a JSON document. | |
Output | Document |
|
| ||
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:
Learn more about Error handling in Pipelines. |
...
Field Name | Field Type | Field Dependency | Description | |
---|---|---|---|---|
Label* Default Value: S3 Browser | String | N/A | The Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline. | |
Bucket Default Value: N/A
| String/Expression/Suggestion | N/A | Specify or select the S3 bucket name, from where the S3 objects are to be listed. A bucket name can be selected from the suggested list. Bucket names are unique globally and can be accessed without its region name in most cases. If you cannot access a bucket name you use cannot be accessed without its region name, the region information can be specified as in the following syntax:
<endpoint> also applies to VPC (Virtual Private Cloud) S3 endpoint. Note that, when : When you enter a wrong region name is given, and if the bucket name is valid, AWS S3 service may successfully access the bucket without any error.
| |
Prefix Default Value: [None]
| IntegerString/Expression/Suggestion | N/A | Specify the prefix of S3 object key names. If Bucket and Prefix are given and all other properties are at default values, the Snap lists all S3 objects matching the prefix in the bucket. An object key can be selected from the suggested list and edited for the prefix. The maximum length of the suggested list is 1,000. If Prefix is expression-enabled, it can be evaluated from an input header document as well as a pipeline parameter. | |
Filter Default Value: [None]**
| Integer/Expression | N/A | Specify the filter the Snap should use internally to list all S3 objects first matching given bucket and prefix. Filter is applied to the entire string of each S3 object key. Filter is a glob pattern which is defined as following:
| |
Show Advanced Properties | CheckboxN/A | Displays Advanced Properties when you select this checkbox. | Select this checkbox to display the advanced properties. Deselect the checkbox to hide the properties. | |
Maximum Object Keys Default Value: All | Integer/Expression | Appears on selecting Show Advanced Properties checkbox. | Maximum number of object keys at the output view. ‘All’ or an empty field produces all matching object key names. | |
Examples: 2500Ignore Empty Result Default Value: All Selected | Ignore Empty Result | Checkbox | Appears on selecting Show Advanced Properties checkbox. | If selected and there is no matching S3 object key, the Snap produces no output document, and if not selected, an empty document. Default Value: Selected |
Maximum Retries* Default Value: 3 | Integer/Expression | Appears on selecting Show Advanced Properties checkbox. | Specify the maximum number of retry attempts to to perform in case of a temporary network loss. | |
Pass Through Default Value: Not Selected Deselected | Checkbox | Appears on selecting Show Advanced Properties checkbox. | Select this checkbox to pass the input document to the output view as a value to the ' | |
Delimiter
| Integer/Expression | Appears on selecting Show Advanced Properties checkbox. | Optional delimiter that causes to produce object keys that contain the same string between the prefix and the first occurrence of the delimiter. For example, a delimiter '/' causes the Snap to produce a non-recursive object-key listing. | |
Snap Execution Default Value: | Dropdown listN/A | Select one of the three modes in which the Snap executes. Available options are:
|
Troubleshooting
Error | Reason | Resolution |
---|---|---|
Account validation failed. | The Pipeline ended before the batch could complete execution due to a connection error. | Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open. |
Examples
Excluding Fields from the Input Data Stream
...