In this article

Overview

The AutoPrep Snap enables you to define simple data transformations without writing expressions. You can use AutoPrep to prepare data for analysis, reporting, and machine learning.

AutoPrep supports the following transformations:

• Flattening leaf nodes of hierarchical data structures

• Removing fields

• Changing the data type of String, Date, Integer, Number, and Boolean fields

• Handling null values

This page describes how to use AutoPrep. AutoPrep Example provides a sample JSON file and walks you through use of AutoPrep.

Snap Type

The AutoPrep Snap is a Transform-type Snap.

Prerequisites

• The preceding Snap is a valid connector

• The preceding Snap outputs data in JSON format

Support for Ultra Pipelines

• Available for Ultra Pipelines

Limitations

• You cannot modify the AutoPrep Snap Label.

• This Snap does not support views or error handling.

• AutoPrep can flatten leaf nodes, but cannot flatten objects.

Using AutoPrep

To use the AutoPrep Snap:

1. In Designer, create a Pipeline to handle the data you want to transform.
   The Snap that precedes AutoPrep must have a document output, not binary.

2. Add the AutoPrep Snap.

3. Click the AutoPrep Snap to open it.
   The left pane has three tabs with controls for preparing data and the right pane displays the data preview in table format:
   

   

The AutoPrep interface provides the following elements and controls:

• Manage fields tab:

  • Flatten Structure tree: Search for fields, and select leaf nodes to flatten them to the root level.

  • Select fields control: Remove fields and change data types.

• Handle Nulls tab: Displays default rules for handling null or empty values. View and modify null handling rules.

• Review Summary tab: Lists the applied changes. Undo or modify changes.

• Preview Data pane: Displays a preview of the data set in a table. As you apply changes, the table updates.

The buttons provide the following functionality:

• Update saves your changes, updates the Preview Data pane, and validates the Pipeline.

• Generate saves your changes and exits AutoPrep. AutoPrep automatically generates the expressions necessary to accomplish the transformations at runtime.

• Cancel exits AutoPrep without saving the current changes.

Flattening a Nested Structure

Flatten the hierarchy using controls on the Manage fields tab. Fields at the root level cannot be flattened.

To flatten a field:

1. In the Flatten fields tree, select the checkbox next to the field or click to select all fields. Non-nested field names are grayed out and you cannot select them.

2. Click Update.

Removing a field

Remove a field using controls on the Manage fields tab:

1. At the bottom of the Manage fields tab, click the Select fields arrow to open the list of fields:

   

2. Click to remove a field. The pill changes from shaded to white:

   

3. Click Update. AutoPrep removes the table column for that field from the Preview data pane.

Changing the Data Type

You can change the data type of a field using controls on the Manage fields tab or in the Preview Data pane. You can only change the data type of a nested field from the Manage fields tab.

To change a data type from the Manage fields tab:

1. Click the Select fields arrow to open the list of fields:

   

2. Click the data type decorator:

   

3. From the menu, select a new data type:

   

4. Change the data types for other fields as required and click Update.
   The updated data types display next to the field names in the Preview data pane.

Handling Nulls

AutoPrep suggests how to handle null values and missing key/value pairs. For example, it suggests Empty string for String fields. If you do not change the null handling, at runtime, when a row includes a null, AutoPrep replaces it with an empty string.

The Manage null values display lists the suggested null handling to the right of each field as shown below:

AutoPrep uses the suggested values unless you explicitly change them. Roll your cursor over the rule to see the others available for that field. You can choose from the following null handling rules:

• For a Boolean field: False, True, Custom input, or Ignore

• For numeric types: Average, Zero, Ignore, or Custom input

• For a String or a Date field: Empty string, Ignore, Custom input, or Popular

When parsing the sample data, AutoPrep detects existing null values and adds a tooltip:

View the null handling rules or modify them as follows:

1. Select the Handle nulls tab to view the Manage null values table:

   

2. To change the rule, roll your cursor over the pill and select from the available options:

   

3. Click Update.

Reviewing and Undoing Changes

The Review summary tab lists all of your formatting and transformation changes. To review, undo, or edit a change:

1. Click Review summary. Each modified field displays with the description of the change indented below it:

   

2. Click to delete a change or to edit it.

Exiting AutoPrep and Previewing Changes

When you are ready to exit AutoPrep, click Generate at the bottom of the screen. If Pipeline validation is not disabled for the Org and Auto Validate is enabled in user settings, you can preview how AutoPrep will prepare the data set at runtime. On the right side of the AutoPrep snap, click to display the data preview:

Troubleshooting

This section describes AutoPrep warnings and error messages.

NaN

If you change the data type of a field to a type that does not support field data, the Preview Data shows as NaN to indicate that the data is not valid for that type:

Warning Icon

When you click Generate, the AutoPrep Snap validates the data set and retains the transformations that you defined. If the structure of the upstream data changes and you re-open AutoPrep, the Review summary warns about those changes.

For example, after AutoPrep Snap validation, if a field was removed from upstream data and you reopen the AutoPrep Snap, the Review summary shows the original column with a warning icon and the Preview Data shows the column as being of an unknown type:

You can remove the field from the Review summary if it was a deliberate deletion.

Error Messages

The following table describes AutoPrep error messages:

Snap Pack History

Related Content

• Mapper Snap

• SnapLogic Glossary

• Getting Started with SnapLogic

• Snap Support for Ultra Pipelines