Skip to end of banner
Go to start of banner

AutoPrep

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 47 Next »

In this article

Overview

The AutoPrep Snap enables you to define simple data transformations without writing expressions. You can use AutoPrep to prepare data for analysis, reporting, and machine learning.

AutoPrep supports the following transformations:

  • Flattening leaf nodes of hierarchical data structures

  • Removing fields

  • Changing the data type of String, Date, Integer, Number, and Boolean fields

  • Handling null values

Snap Type

The AutoPrep Snap is a Transform-type Snap.

The AutoPrep Snap acts as a data preparation application and does not have the same configuration dialogs as other Snaps in the Transform Snap Pack.

Prerequisites

  • The preceding Snap is a valid connector

  • The preceding Snap outputs data in JSON format

Support for Ultra Pipelines

Limitations

  • You cannot modify the AutoPrep Snap Label.

  • This Snap does not support views or error handling.

  • AutoPrep can flatten leaf nodes, but cannot flatten objects.

Using AutoPrep

To use the AutoPrep Snap:

  1. In Designer, create a Pipeline to handle the data you want to transform.
    The Snap that precedes AutoPrep must have a document output, not binary.

  2. Add the AutoPrep Snap.

  3. Click the AutoPrep Snap to open it.
    The left pane has three tabs with controls for preparing data and the right pane displays the data preview in table format:

    AutoPrep interface

The AutoPrep interface provides the following elements and controls:

  • Manage fields tab:

    • Flatten Structure tree: Search for fields, and select leaf nodes to flatten them to the root level.

    • Select fields control: Remove fields and change data types.

  • Handle Nulls tab: Displays default rules for handling null or empty values. View and modify null handling rules.

  • Review Summary tab: Lists the applied changes. Undo or modify changes.

  • Preview Data pane: Displays a preview of the data set in a table. As you apply changes, the table updates.

The buttons provide the following functionality:

  • Update saves your changes, updates the Preview Data pane, and validates the Pipeline.

  • Generate saves your changes and exits AutoPrep. AutoPrep automatically generates the expressions necessary to accomplish the transformations at runtime.

  • Cancel exits AutoPrep without saving the current changes.

Flattening a Nested Structure

Flatten the hierarchy using controls on the Manage fields tab. Fields at the root level cannot be flattened.

To flatten a field:

  1. In the Flatten fields tree, select the checkbox next to the field or click (blue star) to select all fields. Non-nested field names are grayed out and you cannot select them.

  2. Click Update.

Removing a field

Remove a field using controls on the Manage fields tab:

  1. At the bottom of the Manage fields tab, click the Select fields arrow to open the list of fields:

  2. Click (blue star) to remove a field. The pill changes from shaded to white:

  3. Click Update. AutoPrep removes the table column for that field from the Preview data pane.

Changing the Data Type

You can change the data type of a field using controls on the Manage fields tab or in the Preview Data pane. You can only change the data type of a nested field from the Manage fields tab.

The types available depend on the original data type. Object or List data types cannot be converted to other data types nor can simple data types be converted to Object or List.

To change a data type from the Manage fields tab:

  1. Click the Select fields arrow to open the list of fields:

  2. Click the data type decorator:

    Data type decorator
  3. From the menu, select a new data type:

    Data type menu
  4. Change the data types for other fields as required and click Update.
    The updated data types display next to the field names in the Preview data pane.

Handling Nulls

AutoPrep suggests how to handle null values and missing key/value pairs. For example, it suggests Empty string for String fields. If you do not change the null handling, at runtime, when a row includes a null, AutoPrep replaces it with an empty string.

The Manage null values display lists the suggested null handling to the right of each field as shown below:

Null handling rules

AutoPrep uses the suggested values unless you explicitly change them. Roll your cursor over the rule to see the others available for that field. You can choose from the following null handling rules:

  • For a Boolean field: False, True, Custom input, or Ignore

  • For numeric types: Average, Zero, Ignore, or Custom input

  • For a String or a Date field: Empty string, Ignore, Custom input, or Popular

Custom input enables you to specify a value to use for nulls. If you add a value that is not valid for the field type, the null data will output as NaN. Popular causes AutoPrep to calculate and insert the field’s most frequently-used value.

When parsing the sample data, AutoPrep detects existing null values and adds a tooltip:

Null values detected

View the null handling rules or modify them as follows:

  1. Select the Handle nulls tab to view the Manage null values table:

  2. To change the rule, roll your cursor over the pill and select from the available options:

  3. Click Update.

Reviewing and Undoing Changes

The Review summary tab lists all of your formatting and transformation changes. To review, undo, or edit a change:

  1. Click Review summary. Each modified field displays with the description of the change indented below it:

  2. Click (blue star) to delete a change or (blue star) to edit it.

Exiting AutoPrep and Previewing Changes

When you are ready to exit AutoPrep, click Generate at the bottom of the screen. If Pipeline validation is not disabled for the Org and Auto Validate is enabled in user settings, you can preview how AutoPrep will prepare the data set at runtime. On the right side of the AutoPrep snap, click (blue star) to display the data preview:

Preview data

Troubleshooting

This section describes AutoPrep warnings and error messages.

NaN

If you change the data type of a field to a type that does not support field data, the Preview Data shows as NaN to indicate that the data is not valid for that type:

NaN

Warning Icon

When you click Generate, the AutoPrep Snap validates the data set and retains the transformations that you defined. If the structure of the upstream data changes and you re-open AutoPrep, the Review summary warns about those changes.

For example, after AutoPrep Snap validation, if a field was removed from upstream data and you reopen the AutoPrep Snap, the Review summary shows the original column with a warning icon and the Preview Data shows the column as being of an unknown type:

Deleted Column

You can remove the field from the Review summary if it was a deliberate deletion.

Error Messages

The following table describes AutoPrep error messages:

Error

Reason

Resolution

Looks like we couldn’t find a connector here. Please add a connector before you use the AutoPrep experience.

An upstream Snap must provide the sample data for AutoPrep.

Add a valid Snap that outputs JSON before the AutoPrep Snap.

We couldn’t find any preview data. Please try running the validation again before we can get to AutoPrepping.

The upstream Snap is not valid.

Make sure the data source is connected and that the upstream Snap is outputting JSON.


Snap Pack History

Related Content

  • No labels