Versions Compared

Version Old Version 8 New Version Current
Changes made by

Cindy Hall (Unlicensed)

Kalpana Malladi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

Table of Contents
minLevel1
maxLevel2
absoluteUrltrue

Overview

You can use this Snap to archive an S3 object in the S3 bucket. Apart Aside from archiving S3 objects, you can also change the storage class from the current class to another if the transition meets the requirements of AWS S3.

AWS S3 does not define any folder object and the '/' characters are part of the S3 object key names.

...

Snap Type

The S3 Archive Snap is a Write-type Snap.

Prerequisites

  • A valid account with the required permissions.
    [I think this is required. However, Paul listed None.]

Support for Ultra Pipelines

Limitations

None.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 0

  • Max: 1

  • Mapper

  • S3 Upload

  • JSON Formatter

An upstream Snap is optional. Any document with key-value pairs to evaluate expression properties. Each input document, if any, results in changing the storage class of an S3 object.

Output

Document

  • Min: 1

  • Max: 1

  • Mapper

  • S3 Restore

  • S3 Download

S3 Archive Snap output in JSON formatImage Removed
S3 Archive Snap output in JSON formatImage Added

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

Info

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the fieldset.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the fieldset.

  • Upload icon ((blue star) ): Indicates that you can upload files. //is this missing by design?

Field Name

Field Type

Field Dependency

Description

Label*

 

Default Value: S3 Archive
Example:

 Project

 Project Archive

String

None

N/A

 

Specify a unique name for the Snap.

 

Label*

 

Default Value: S3 Archive
Example: Project Archive

You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Bucket*

 

Default Value: None
Example:

String/Expression/Suggestion

None

N/A

 

Specify the S3 bucket name where the S3 object to be archived is located.

Note

Do not add S3:/// before the bucket name, because the Snap can fail.

  • Bucket names are unique globally and can be accessed without

its

  • the region name in most cases. If you cannot access a bucket name without its region name, you can specify the region information

in

  • using the following syntax:

    • <S3_bucket_name>@<region_name>

  • You can also access the S3 bucket in an S3 Virtual Private Cloud (VPC) endpoint by specifying the bucket name

in

  • using the following syntax:

    • <S3_bucket_name>@<VPC_S3_endpoint>

Note:

When

If you enter an incorrect region name, but

if

the bucket name is valid, the AWS S3 service may successfully access the bucket without any

error.

 

Bucket*

 

Default Value: None
Example:

errors.

Object Key*

 

Default Value: None
Examples: 

  • test.csv

  • abc/test.json

  • abc/xyz/test.xml

String/Expression/Suggestion

N/A

None

Specify the S3 object key name, which may include one or more forward-slash ('/') characters.

The forward-slash character is part of the S3 object key name and there is no folder object defined in AWS S3. The maximum length of the suggested list is 1,000.

 

Object Key*

 

Default Value: None

Examples:

 

  • test.csv

  • abc/test.json

    • abc/xyz/test.xmlChange Storage Class

    Change Storage Class

    Composite

    N/A

    Use this field set to change the storage class of S3 objects.

    From

    Default Value: None
    Examples: 

    Standard

    String

    /Expression

    /Suggestion

    N/A

    None

    This field is for information only and not used when the Snap is previewed or executed. The current storage class of the selected S3 object can selected from the suggestion.

    From

    Default Value: None
    Examples: 

    • Standard

    To

    Default Value: None
    Examples: 

    • GlacierInstantRetrieval

    • GlacierFlexibleRetrieval

    • DeepArchive

    String/Expression/Suggestion

    N/A

    None

    Enter or select the target storage class name from the suggested list.

    Supported storage classes are:

    • Standard

    • GlacierInstantRetrieval

    • GlacierFlexibleRetrieval

    • DeepArchive

    • IntelligentTiering

    • StandardInfrequentAccess

    • OneZoneInfrequentAccess

    • ReducedRedundancy

    To

    Default Value: None

    • GlacierInstantRetrieval

    • GlacierFlexibleRetrieval

    • DeepArchive

    Examples: 

    Show Advanced Properties

     

    Default Value: Deselected
    Example: Selected

    Checkbox

    N/A

    None

    Select this checkbox to display the advanced properties.
    Deselect this checkbox to hide the properties.

    Show Advanced Properties

     

    Default Value: Deselected
    Example: Selected

    Thread Pool Size

     

    Default Value: 10
    Example: 20

    Integer/Expression

    Appears when you select the Show Advanced Properties checkbox.

    Specify the maximum number of threads to be used to archive multiple S3 objects in parallel with Bucket and Object Key properties expression-enabled.

    If the Thread Pool Size property is expression-enabled, it can be evaluated with a Pipeline parameter, but not with an input document. Therefore, the value of this property remains constant throughout the Pipeline execution.

    Thread Pool Size

     

    Default Value: 10
    Example: 20

    Maximum Retries*

     

    Default Value: 3
    Example: 2

    Integer/Expression

    Appears when you select the Show Advanced Properties checkbox.

    Specify the maximum number of retry attempts to perform in case of a temporary network loss.

     

    Maximum Retries*

     

    Default Value: 3
    Example: 2

    Pass Through

     

    Default Value: Deselected
    Example: Selected

    Checkbox

    Appears when you select the Show Advanced Properties checkbox.

    Select this checkbox to pass the input document to the output view as a value to the 'original' key .

    Pass Through

     

    Default Value: Deselected
    Example: Selected

    Snap Execution

    Default Value: Execute only
    Example: Validate & Execute

    Dropdown list

    N/A

    None

    Select one of the following three modes in which the Snap executes

    . Available options are

    :

    • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

    • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

    • Disabled: Disables the Snap and all Snaps that are downstream from it.

    Snap Execution

    Default Value: 
    Example: Validate & Execute

    Number of records

    Default Value: 
    Example:

    String/Expression

    Sampling Type is Number of records.

    Enter the number of records to output.

    Field set Name

    Specify advanced parameters that you want to include in the request.

    Field 1*

    Default Value: <value> or None.
    Example: <value>

    String

    Debug mode check box is not selected.

    Field 2

    Default Value: <value> or None.
    Example: <value>

    String

    None.

    Snap Execution

    Default Value: 
    Example: Validate & Execute

    Dropdown list

    N/A

    Select one of the following three modes in which the Snap executes:

    • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

    • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

    • Disabled: Disables the Snap and all Snaps that are downstream from it.

    Troubleshooting

    ...

    Error

    ...

    Reason

    ...

    Resolution

    ...

    Account validation failed.

    ...

    The Pipeline ended before the batch could complete execution due to a connection error.

    ...

    Verify that the Refresh token field is configured to handle the inputs properly. If you are not sure when the input data is available, configure this field as zero to keep the connection always open.

    Examples

    Excluding Fields from the Input Data Stream

    We can exclude the unrequired fields from the input data stream by omitting them in the Input schema field set. This example demonstrates how we can use the <Snap Name> to achieve this result:

    ...

    Examples

    Browsing and Archiving Amazon S3 Objects

    This example Pipeline demonstrates how to browse an S3 object and then archive it.

    ...

    Prerequisites: A valid AWS S3 Account

    1. Configure the S3 Browser Snap to browse the test/test100.json object from the mrtest bucket.

      S3 Browser settingsImage Added

    2. Validate the Pipeline. The output displays the details of the browsed S3 object. Note that the storage class (Standard) is eligible for archiving.

      S3 Browser output details in JSON formatImage Added

    3. Next, configure the S3 Archive Snap to archive the S3 object. In this example we are changing the storage class from Standard to GlacierFlexibleRetrieval.

      S3 Archive Snap settings exampleImage Added

    4. Validate the Pipeline. The output displays the details of the archived S3 object. You can verify that the storage class is changed.

      S3 Archive Ouput viewImage Added

    Download this Pipeline. 

    Downloads

    Info

    1. Download and import the Pipeline into SnapLogic.

    2. Configure Snap accounts, as applicable.

    3. Provide Pipeline parameters, as applicable.

    Attachments
    patterns*.slp, *.zip
    sortByname

    Snap Pack History

    Insert excerpt
    XYZ Amazon S3 Snap PackXYZ
    Amazon S3 Snap Pack

    ...

    Endpoint Doc Link 1

    ...

    Endpoint Doc Link 2

    ...

    Endpoint Doc Link 3

    ...

    SnapLogic Glossary

    ...

    Getting Started with SnapLogic

    ...

    nameS3 Snap Pack History
    nopaneltrue

    ...