S3 Archive

In this article

Overview

You can use this Snap to archive an S3 object in the S3 bucket. Aside from archiving S3 objects, you can also change the storage class from the current class to another if the transition meets the requirements of AWS S3.

AWS S3 does not define any folder object and the '/' characters are part of the S3 object key names.

S3 Archive Snap settings

Snap Type

The S3 Archive Snap is a Write-type Snap.

Prerequisites

  • A valid account with the required permissions.

Support for Ultra Pipelines

Limitations

None.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

 

  • Min: 0

  • Max: 1

  • Mapper

  • S3 Upload

  • JSON Formatter

An upstream Snap is optional. Any document with key-value pairs to evaluate expression properties. Each input document, if any, results in changing the storage class of an S3 object.

Output

Document

 

  • Min: 1

  • Max: 1

  • Mapper

  • S3 Restore

  • S3 Download

S3 Archive Snap output in JSON format

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( ): Indicates that you can add fields in the fieldset.

  • Remove icon ( ): Indicates that you can remove fields from the fieldset.

  • Upload icon ( ): Indicates that you can upload files.

Field Name

Field Type

Field Dependency

Description

Field Name

Field Type

Field Dependency

Description

Label*

 

Default ValueS3 Archive
ExampleProject Archive

String

None

 

Specify a unique name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Bucket*

 

Default Value: None
Example:

String/Expression/Suggestion

None

 

 

Specify the S3 bucket name where the S3 object to be archived is located.

Do not add S3:/// before the bucket name, because the Snap can fail.

Object Key*

 

Default Value: None
Examples

  • test.csv

  • abc/test.json

  • abc/xyz/test.xml

String/Expression/Suggestion

None

 

Specify the S3 object key name, which may include one or more forward-slash ('/') characters.

 

Change Storage Class

Use this field set to change the storage class of S3 objects.

From

Default Value: None
ExamplesStandard

String/Suggestion

None

This field is for information only and not used when the Snap is previewed or executed. The current storage class of the selected S3 object can selected from the suggestion.

To

Default Value: None
Examples

  • GlacierInstantRetrieval

  • GlacierFlexibleRetrieval

  • DeepArchive

String/Expression/Suggestion

None

Enter or select the target storage class name from the suggested list.

Supported storage classes are:

  • Standard

  • GlacierInstantRetrieval

  • GlacierFlexibleRetrieval

  • DeepArchive

  • IntelligentTiering

  • StandardInfrequentAccess

  • OneZoneInfrequentAccess

  • ReducedRedundancy

Show Advanced Properties

 

Default Value: Deselected
Example: Selected

Checkbox

None

Select this checkbox to display the advanced properties.
Deselect this checkbox to hide the properties.

Thread Pool Size

 

Default Value: 10
Example: 20

Integer/Expression

Appears when you select the Show Advanced Properties checkbox.

Specify the maximum number of threads to be used to archive multiple S3 objects in parallel with Bucket and Object Key properties expression-enabled.

Maximum Retries*

 

Default Value: 3
Example: 2

Integer/Expression

Appears when you select the Show Advanced Properties checkbox.

Specify the maximum number of retry attempts to perform in case of a temporary network loss.

 

Pass Through

 

Default Value: Deselected
Example: Selected

Checkbox

Appears when you select the Show Advanced Properties checkbox.

Select this checkbox to pass the input document to the output view as a value to the 'original' key .

Snap Execution

Default ValueExecute only
Example: Validate & Execute

Dropdown list

None

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Examples

Browsing and Archiving Amazon S3 Objects

This example Pipeline demonstrates how to browse an S3 object and then archive it.

Prerequisites: A valid AWS S3 Account

  1. Configure the S3 Browser Snap to browse the test/test100.json object from the mrtest bucket.

  2. Validate the Pipeline. The output displays the details of the browsed S3 object. Note that the storage class (Standard) is eligible for archiving.

     

  3. Next, configure the S3 Archive Snap to archive the S3 object. In this example we are changing the storage class from Standard to GlacierFlexibleRetrieval.

     

  4. Validate the Pipeline. The output displays the details of the archived S3 object. You can verify that the storage class is changed.

     

Download this Pipeline

Downloads

 

  File Modified

File Example_S3Archive.slp

Oct 04, 2022 by Cindy Hall

Snap Pack History


Related Links