...
Table of Contents | ||||||
---|---|---|---|---|---|---|
|
Overview
You can use the Adobe Experience Platform S3 Connector Snap to read files from AWS S3 and write them as JSON or Parquet files into Adobe Experience Platform (AEP). The AWS file that you want to read should have the same structure as the schema specified in the Schema Name field. If you do not have a file configured appropriately, use the File Generator Snap to generate the file and then reference it here.
Prerequisites
The schema that you want to use with this Snap should exist in Adobe Experience Platform. If you do not have the schema ready, create it using the Adobe Experience Platform user interface before using this Snap.
Limitations
None.
File Delete Snap retrieves attributes of contents of a given directory.
Snap Type
File Delete Snap is a read type snap that retrieves attributes of contents of a given directory.
Prerequisites
None
Account
This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint.
Limitations and Known Issues
None.
Snap
...
Input/Output
...
Type of View
...
Number of Views
...
Compatible Upstream and Downstream Snaps
Views
Views | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
Input | Document |
|
| Each input document contains details that can be used with the Schema Name, Dataset Name, or Connector Object Name fields to read data from AWS S3 and write them into AEP. | Output | Document |
|
| Each output document lists out the status and other details associated with the AEP write processThis Snap has at most one document input view. Each document contains data in the form of json. |
Output | This Snap has at most one document output view. Each document contains a corresponding response in the form of JSON. | ||||||||
Error | This Snap has at most one document error view and produces zero or more documents in the view. Each document contains the corresponding error record if snap is not able to process data to Custom SFTP. |
Snap Settings
Error
Reason
Resolution
Invalid Snap configuration
Property Schema Name is null or missing
Provide a valid schema name.
Unable to load the private key for the given alias
File not found on <your SnapLogic instance> at <account location>
This error appears when the account doesn't have valid values. Ensure that the keystore path, keystore passphrase, private key alias, and private key passphrase are correct. Also, you must upload the keystore in the Keystore Path that you specify.
Unable to obtain access token
This message appears when the endpoint account details are incorrect.
Parameter Name | Data Type | DescriptionDefault Value | ||||
---|---|---|---|---|---|---|
Label | Example required | Label
| String | Required. The name for the Snap. Modify You can modify this to be more specific, especially if there is you have more than one of the same Snap in the Pipeline. | Adobe Experience Platform S3 Connector | Write Documents from S3 to AEP |
Schema Name | String | Required. The name of the schema to be used to create the dataset to be written into AEP. This is a suggestible field. Click to view schema suggestions that you can use. | N/A | Snaplogic Demo Experience Event Schema | ||
Dataset Name | String | Required. The name of the dataset into which you want to write the AWS S3 data. The Snap automatically creates a new dataset if a dataset with the specified name does not exist.
This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties.
| N/A | Adobe Generic Dataset | ||
Connectors Object Name | String | Required. The URL of the AWS S3 object that you want to read.
This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties.
Protocol required for this Snap: S3://. | s3:// | s3://testing-sl-bucket/xdm/File_Pipeline_3000000.parquet | String | Required. The schema type that you want to use to create the file that must be written. Click to select from the following options:
The schema type of the source and destination files must be the same. | Parquet | Parquet |
Batch Monitor Tries | Integer | Required. The number of times you want a batch to be monitored until it is completed. | 360 | 300 | ||
Snap Execution | String | Select one of the three modes in which the Snap executesyour pipeline. Default value: [None] | ||||
Directory required | The URL for the data source (directory). It should start with a protocol. | |||||
File filter Conditionally required
| The GLOB pattern can be applied to select the contents (files/sub-folders) of the directory. You cannot recursively navigate through the directory structures. The File filter property can be a JavaScript expression, which will be evaluated with the values from the input view document. Example:
Default value: [None] | |||||
Ignore empty result Conditionally required
| If selected, no document will be written to the output view when the result is empty. If this property is not selected and the Snap has received an input document, the input document will be passed through to the output view. If this property is not selected and there is no input document, an empty document will be written to the output view. Default value: Selected | |||||
Display update time and owner for S3 subdirectories Conditionally required
| Applicable only for S3 directory. If selected, the Snap produces Update time and Owner attributes in the output documents for S3 subdirectories. However, it may take a long time or even freeze if there are very large number of files and subdirectories in the file structure under the specified S3 directory. This is because the Snap has to completely examine the S3 objects under the directory recursively in order to get Update time and Owner attributes for subdirectories. If not selected, the Snap does not produce Update time and Owner attributes in the output documents for S3 subdirectories. Snap output is faster since it examines only S3 files and subdirectories immediately under the specified S3 directory. Default value: Selected | |||||
Number of retries optional | Specifies the maximum number of retry attempts in case of a network failure. Example: 3 Minimum value: 0 Default value: 0 | |||||
Retry interval (seconds) | Specifies the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure. Example: 3 Minimum value: 1 Default value: 1 | |||||
Advanced properties | Use this fieldset to define specific settings for directory browsing. Click + to add a new row for defining an advanced property. This fieldset comprises the following fields:
| |||||
Snap Execution | Indicates how the Snap must be executed. Available options are:
| Execute only | Validate and Execute |
Troubleshooting
Default value: Validate & Execute |