Adobe Experience Platform S3 Connector

In this article

Overview

Use the Adobe Experience Platform S3 Connector Snap to read files from AWS S3 and write them as JSON or Parquet files into Adobe Experience Platform (AEP). The AWS file that you want to read should have the same structure as the schema specified in the Schema Name field. If you do not have a file configured appropriately, use the File Generator Snap to generate the file and then reference it here.

Prerequisites

  • The schema that you want to use with this Snap should exist in Adobe Experience Platform. If you do not have the schema ready, create it using the Adobe Experience Platform user interface before using this Snap.

Limitations and Known Issues

  • None.

Snap Input and Output

Input/OutputType of ViewNumber of ViewsCompatible Upstream and Downstream SnapsDescription
Input Document
  • Min: 0
  • Max: 1
  • Mapper Snap
  • Binary to Document Snap
  • Sort Snap
Each input document contains details that can be used with the Schema Name, Dataset Name, or Connector Object Name fields to read data from AWS S3 and write them into AEP.
OutputDocument
  • Min: 0
  • Max: 1
  • Mapper Snap
  • Join Snap
  • Union Snap

Each output document lists out the status and other details associated with the AEP write process.

Snap Settings

Parameter NameData TypeDescriptionDefault ValueExample 
LabelStringRequired. The name for the Snap. Modify this to be more specific, especially if there is more than one of the same Snap in the Pipeline.Adobe Experience Platform S3 ConnectorWrite Documents from S3 to AEP
Schema NameString

Required. The name of the schema to be used to create the dataset to be written into AEP. This is a suggestible field. Click  to view schema suggestions that you can use.

N/ASnaplogic Demo Experience Event Schema

Dataset Name

String

Required. The name of the dataset into which you want to write the AWS S3 data. 

The Snap automatically creates a new dataset if a dataset with the specified name does not exist.

This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties.

N/AAdobe Generic Dataset
Connectors Object NameString

Required. The URL of the AWS S3 object that you want to read.

This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties.

Protocol required for this Snap: S3://.

s3://s3://testing-sl-bucket/xdm/File_Pipeline_3000000.parquet

String

Required. The schema type that you want to use to create the file that must be written. Click to select from the following options:

  • Parquet
  • JSON

The schema type of the source and destination files must be the same. 

ParquetParquet
Batch Monitor TriesInteger

Required. The number of times you want a batch to be monitored until it is completed.

360300
Snap execution StringSelect one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Execute onlyValidate and Execute

Troubleshooting

ErrorReasonResolution
Invalid Snap configurationProperty Schema Name is null or missingProvide a valid schema name.
Unable to load the private key for the given aliasFile not found on <your SnapLogic instance> at <account location>This error appears when the account doesn't have valid values. Ensure that the keystore path, keystore passphrase, private key alias, and private key passphrase are correct. Also, you must upload the keystore in the Keystore Path that you specify.
Unable to obtain access tokenThis message appears when the endpoint account details are incorrect.Check your Organization ID, Technical Account ID, Client ID, Client Secret Key details and try again.

Examples

Reading Parquet Documents from S3 and Writing them into AEP

In this example, you retrieve a Parquet document from your AWS S3 account and write it into a target AEP dataset. 

For this example, you only need an Adobe Experience Platform S3 Connector Snap. You open the Snap and configure it to retrieve the file that you want from AWS S3 and write it into AEP.

  1. Configure your Adobe Experience Platform account in the Account tab.
  2. Click the  field adjacent to the Schema Name field and select the schema that you want to use for the file coming in from AWS S3.
  3. Enter the name of the dataset that you want to create, in the Dataset Name field.
  4. Enter the URL of the source file in AWS S3, in the Connectors Object Name field.
  5. Select the Schema Type that you want to use.
  6. Use the Batch Monitor Tries field to specify the number of times you want the batch to be monitored until it is completed.
  7. Validate your settings and execute the Pipeline. 

    If your settings are correct, the Snap executes successfully, and displays details associated with the operation.

Download this Pipeline

Downloads

  File Modified

File AEP_S3Connector_Example.slp

Apr 24, 2020 by Siddharth Menon

Snap Pack History

 Click to view/expand

4.24 (main8556)

  • No updates made.

4.23 423patches7447

  • Fixes the Adobe Experience Platform Execute Snap issue of not passing data correctly to the downstream Snap by creating Map obj = new LinkedHashMap(); inside loop. LinkedHashMap is a hash table and linked list implementation of the Map interface, with a predictable iteration order. 

4.23 (main7430)

  • Replaces the Adobe Cloud Platform Snap Pack.
  • Introduces the Adobe Experience Platform Read Snap that enables executing SQL queries in the Adobe Experience Platform.
  • Updates the Adobe Experience Platform JWT Account to enable you to create accounts in a Sandbox location during the development phase.
  • Enhances the Adobe Experience Platform Read Snap by adding fields, Batch IDsStart date, and End Date to allow filtering batches by ID, start date, and end date with millisecond precision.

4.22 (main6403)

4.21 Patches

4.21 (snapsmrc542)

4.20 (snapsmrc535)

  • No updates made.

4.19 (snapsmrc528)

  • No updates made.

4.18 (snapsmrc523)

  • Enhanced the Snap Pack to support the Map datatype in XSD-based datasets.

4.17 Patch ALL7402

  • Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17 (snapsmrc515)

  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16 (snapsmrc508)

  • No updates made. Automatic rebuild with a platform release.

4.15 (snapsmrc500)

  • No updates made. Automatic rebuild with a platform release.

4.14 (snapsmrc490) 

  • No updates made. Automatic rebuild with a platform release.

4.13 (snapsmrc486)

  • New! Initial release of the Snap Pack. This includes Adobe Cloud Platform Read and Adobe Cloud Platform Write Snaps.
  • Introduced the basic Adobe Cloud Platform JWT Account.