$customHeader
Skip to end of banner
Go to start of banner

S3 Restore

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 28 Current »

In this article

Overview

You can use this Snap to restore an archived S3 object in an S3 bucket.

S3 Restore Snap Settings

Snap Type

The S3 Restore Snap is a Write-type Snap.

Prerequisites

  • A valid account with the required permissions.

  • The S3 object should have been archived by changing its storage class to GlacierFlexibleRetrieval or DeepArchive before executing the S3 Restore Snap.

Support for Ultra Pipelines

Works in Ultra Pipelines

Limitations and Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

  • Min: 0

  • Max: 1

  • Mapper

  • JSON Parser

An upstream Snap is optional. Any document with key-value pairs to evaluate expression properties.

Output

Document

  • Min: 0

  • Max: 1

  • Mapper

  • JSON Parser

When Check Status Until Completed is selected in Settings:

Preview data in a table

When Check Status Until Completed is not selected in Settings:

Preview data in a table

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon ((blue star)): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ((blue star) ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( (blue star) ): Indicates that you can add fields in the fieldset.

  • Remove icon ( (blue star)): Indicates that you can remove fields from the fieldset.

  • Upload icon ((blue star) ): Indicates that you can upload files.

Field Name

Field Type

Field Dependency

Description

Label*

Default ValueS3 Restore
ExampleRestore Invoice - March 2018

String

None

The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Bucket*

 

Default Value: None
Example:

  • mybucket

  • mybucket@eu-west-1

  • S3 Bucket in S3 region:

    • mybucket@s3.eu-west-1.amazonaws.com

  • S3 VPC endpoint:

    • my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

    • testBucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

String/Expression/Suggestion

None

 

 

Specify the S3 bucket name where the S3 object to be restored is located.

Do not add S3:/// before the bucket name, because the Snap can fail.

  • Bucket names are unique globally and can be accessed without the region name in most cases. If you cannot access a bucket name without its region name, you can specify the region information using the following syntax:

    • <S3_bucket_name>@<region_name>

  • You can also access the S3 bucket in an S3 Virtual Private Cloud (VPC) endpoint by specifying the bucket name using the following syntax:

    • <S3_bucket_name>@<VPC_S3_endpoint>

Note: If you enter an incorrect region name, but the bucket name is valid, the AWS S3 service may successfully access the bucket without any errors.

Object Key*

Default Value: None
Examples

  • test.csv

  • abc/test.json

  • abc/xyz/test.xml

String/Expression/Suggestion

None

Specify the S3 object key name, which may include one or more forward-slash ('/') characters.

The S3 object should have been archived by changing its storage class to GlacierFlexibleRetrieval or DeepArchive before executing the S3 Restore Snap.

The forward-slash character is part of the S3 object key name and there is no folder object defined in AWS S3. The maximum length of the suggested list is 1,000.

Expiration (Days)

Default Value: 1

Minimum Value: 1
Example: 2

Integer

Expression

None

Enter the lifetime of the active copy of the S3 object in days.

If an S3 object's restoring process is completed, the content of the S3 object can be accessed until its active copy expires.

Retrieval Tier

Default Value: None
Examples:

  • Bulk

  • Standard

  • Expedited

Dropdown list

None

Choose one of the available S3 Glacier retrieval tiers: Bulk, Standard, or Expedited.

Check Status Until Completed

Default Value: Deselected

ExamplesSelected

Checkbox

None

Select this checkbox to poll the job status of restoring an archived S3 object until it is completed.

The restore process may take 5 to 12 hours. When this checkbox is selected, the Snap checks the S3 object restore status continuously in a given interval until the restore is completed.

Check Status Interval (Minutes)

Default Value: 10

Minimum Value: 1
Example30

Integer/Expression

Appears when you select the Check Status Until Completed checkbox.

Enter the minimum amount of time in minutes to pause between the polling of the job status of restoring an archived S3 object.

Show Advanced Properties

Default Value: Deselected
Example: Selected

Checkbox

None

Select this checkbox to display the advanced properties.
Deselect this checkbox to hide the properties.

Thread Pool Size

Default Value: 10
Example: 20

Integer/Expression

Appears when you select the Show Advanced Properties checkbox.

Specify the maximum number of threads to be used to restore multiple S3 objects in parallel with Bucket and Object Key properties expression-enabled.

If the Thread Pool Size property is expression-enabled, it can be evaluated with a Pipeline parameter, but not with an input document. Therefore, the value of this property remains constant throughout the Pipeline execution.

Maximum Retries*

Default Value: 3
Example: 2

Integer/Expression

Appears when you select the Show Advanced Properties checkbox.

Specify the maximum number of retry attempts to perform in case of a temporary network loss.

Pass Through

Default Value: Deselected
Example: Selected

Checkbox

Appears when you select the Show Advanced Properties checkbox.

Select this checkbox to pass the input document to the output view as a value to the 'original' key .

Snap Execution

Default ValueExecute only
Example: Validate & Execute

Dropdown list

None

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Examples

Browsing and Restoring Amazon S3 Objects

This example Pipeline demonstrates how to browse an S3 object and then restore it.

S3 Browser Snap connected to S3 Restore Snap

Prerequisites: A valid AWS S3 Account

  1. Configure the S3 Browser Snap to browse the mrtest/test100.json object from the mrtest bucket.

    S3 Browser configuration settings

  2. Validate the Pipeline. The output displays the details of the browsed S3 object. Note that the storage class (GlacierFlexibleRetrieval) is eligible for restoration.

    S3 Browser output details in JSON format

  3. Next, configure the S3 Restore Snap to restore the S3 object, which is currently archived.

    S3 Restore Snap settings example

  4. Validate the Pipeline. The output displays the details of the restored S3 object, including the restoration status of the object (Restore requested or Restore completed).

    When Check Status Until Completed is not selected:

    Preview data in a table, showing the expiration in days and status (Restore requested)



    When Check Status Until Completed is selected:

    Preview data in a table, showing the expiration in days, status (Restore completed) expiration time, and storage class


Download this Pipeline

Downloads

  1. Download and import the Pipeline into SnapLogic.

  2. Configure Snap accounts, as applicable.

  3. Provide Pipeline parameters, as applicable.

  File Modified
No files shared here yet.

Snap Pack History

Release

Snap Pack Version

Date

Type

Updates

May 2024

main26341

Stable

Enhanced the S3 Select Snap to capture metadata and lineage information from the input document.

February 2024

436patches25360

Latest

Fixed an issue with the Amazon S3 Snaps that displayed a null pointer exception when the Access Key ID or Secret Key field was empty while utilizing the S3 Express Bucket in the S3 Account. The Snaps now throw the configuration exception if either field is empty.

February 2024

main25112

Stable

Updated and certified against the current Snaplogic Platform release.

November 2023

435patches24238

Latest

Added support for Amazon S3 Express One Zone in the Amazon S3 Snap Pack.

November 2023

main23721

Stable

Updated and certified against the current Snaplogic Platform release.

August 2023

main22460

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023

433patches21816

Latest

The Amazon S3 Snaps automatically detect the Maximum session duration value for the Cross-Account IAM role (1 through 12 hours). The Snaps round down the value to the nearest hour. So, if the Snap administrator sets the Maximum session duration at 3 hours and 45 minutes, the Snaps read it as 3 hours. The Snaps also refresh the session before it expires. However, the automatic session refresh does not support the case of very large file upload or download that takes longer than the maximum session duration.

May 2023

main21015

Stable

Upgraded with the latest SnapLogic Platform release.

February 2023

432patches20385

Latest

Added support for Ultra Task Pipelines.

February 2023

main19844

Stable

Upgraded with the latest SnapLogic Platform release.

November 2022

main18944

Stable

  • The S3 Browser Snap output now includes the Storage Class field, which indicates the archived status of the S3 object.

  • The S3 Download Snap no longer fails even when the pipeline has multiple Snaps after 430patches18348.

October 2022

430patches18674

Latest

  • Introduced the following Snaps:

    • S3 Archive enables you to archive an S3 object and change its storage class.

    • S3 Restore enables you to restore an archived S3 object.

    • S3 Select enables you to retrieve a subset of data from an S3 object.

  • The S3 Download, S3 Archive, S3 Copy, S3 Delete, S3 Restore, and S3 Upload Snaps do not have the increased number of active threads accumulated, as they are now released immediately after the execution.

  • The S3 Download Snap now does not fail even when the pipeline has multiple Snaps after 430patches18348.

  • The S3 Browser Snap output now includes the Storage Class field, which indicates the archived status of the S3 object.

August 2022

430patches17354

Latest

The KMS Region field in the S3 Account now suggests the regions when you click the suggestion (blue star) icon.

August 2022

main17386

Stable

Introduced the Amazon S3 Snap Pack, which enables you to browse, copy, delete, download, or upload objects in S3. This Snap Pack contains the following Snaps:

  • S3 Browser: Lists the attributes of Amazon S3 objects in a specific bucket matching the prefix.

  • S3 Copy: Sends a copy request to the AWS S3 service to copy an Amazon S3 object from a source bucket to a target bucket.

  • S3 Delete: Removes an object from the specified bucket.

  • S3 Download: Downloads Amazon S3 objects from the S3 bucket.

  • S3 Upload: Uploads binary data to Amazon S3 objects.

  • S3 Presigned: Generates a presigned URL in the output document to access an Amazon S3 object.


  • No labels