Versions Compared

Version Old Version 42 New Version 43
Changes made by

Cindy Hall

Cindy Hall

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

...

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 0

  • Max: 1

  • Mapper

  • S3 Copy

Key/value pairs used to evaluate expression properties in this Snap. 

Output

Document

  • Min: 1

  • Max: 2

  • S3 Download

  • Mapper

The output displays the following attributes:

  • bucket: The S3 bucket name.

  • objectKey: The S3 object with the prefix.

  • path: The path to the S3 bucket.

  • size: The size of the S3 object.

  • lastModified: The date when the S3 object was last modified.

  • storage-class: The Storage Class field, which indicates the archived status of the S3 object.

  • ownerDisplayName: The display name of the S3 owner.

  • ownerID: The ID of the S3 owner.

The attributes for each S3 object matching the specific Snap field values. For example, if you have the following S3 objects with the prefix "foo/" in the bucket "mrtest":

Info

  • foo/bar/ban

  • foo/bar/bash

  • foo/bar/baz

  • foo/batbat/baz

  • foo/boo

If the S3 Browser Snap has the following properties, then the Snap produces output documents for all the five S3 objects.

  • Bucket: mrtest

  • Prefix:  foo/

  • Filter:   **

  • Delimiter:

If the S3 Browser Snap has the following properties, then the Snap produces only one output document for foo/boo.

  • Bucket: mrtest

  • Prefix:  foo/

  • Filter:   **

  • Delimiter: /

 

Note

When you use the Cross Account IAM Role in the S3 account, the ownerDisplayName and ownerID fields may be empty in the output document.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

...

  1. Configure key-value pairs containing parameters to pass to the Pipeline during execution.

  2. Configure the JSON Generator Snap with sample data as follows.

  3. Configure the JSON Formatter Snap to format the data. You can view the JSON binary data in the output preview of the Snap.

  4. Configure the S3 Upload Snap to upload the binary data to S3 objects residing in the S3 bucket.

  5. Configure the S3 Copy Snap to copy the S3 object from source folder to target folder.

  6. Configure the S3 Browser Snap to list the attributes for S3 objects.

    Image RemovedS3 Browser confiiguration settingsImage Added

  7. Configure the S3 Download Snap to download S3 objects.

  8. As the output is in binary format, you need to convert the binary data to document data using the Binary to Document Snap.

  9. Configure the S3 Delete Snap to delete the S3 objects. You can view the status of the deleted object in the output preview.

...