Aggregate

In this article

Overview

You can use this Snap to apply aggregate functions on input data using the Group By support. This Snap enables you to calculate an aggregate function on a set of values to return a single scalar value.

  • This Snap does not support list and map objects referenced in the JSON paths.

  • If the input documents are unsorted and GROUP-BY fields are used, you must use the Sort Snap upstream of the Aggregate Snap to presort the input document stream and set the Sorted stream field Ascending or Descending to prevent the out-of-memory error. However, if the total size of input documents is expected to be relatively small compared to the available memory, then Sort Snap is not required upstream.
    Learn more about presorting unsorted input documents to be processed by the Aggregate Snap.

The following are the commonly used SQL Aggregate functions:

  • AVG – calculates the average of a set of values.

  • COUNT – counts rows in a specified table or view.

  • MIN – gets the minimum value in a set of values.

  • MAX – gets the maximum value in a set of values.

  • SUM – calculates the sum of values.

  • CONCAT – calculates the sum of values.

  • UNIQUE_CONCAT – calculates the sum of values.

Snap Type

Aggregate Snap is a Transform-type Snap that transforms, parses, cleans, and formats data from binary to document data.

Prerequisites

None.

Support for Ultra Pipelines

Does not support Ultra Pipelines

Limitations and Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

 

  • Min: 1

  • Max: 1

  • JSON

  • Mapper

Each document should contain values referenced in the Aggregate fields and the GROUP-BY fields field set. If not, the input data is sent to the error view.

Output

Document

 

  • Min: 1

  • Max: 1

  • JSON

  • Mapper

Each document contains the mapped data that includes key-value entries of the GROUP-BY field name and its value, and a key-value entry of the Result field and its value, if processed successfully.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution: Stops the current Pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk (*): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon (): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( ): Indicates that you can add fields in the field set.

  • Remove icon (): Indicates that you can remove fields from the field set.

Field Name

Type

Description

Field Name

Type

Description

Label*

Default ValueAggregate
ExampleAggregate_Avg

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

 

Aggregate fields*

 

 

 

Use this field set to define the type of Aggregate function to perform on the field and the key name to be used in the output. This field set contains the following fields:

  • Function

  • Field

  • Result field

Function*

Default ValueSUM
ExampleAVG

String

Select the functions that applies to the aggregate field value in the input data. The available functions and the supported datatypes are: 

  • COUNT – all types.

  • SUM – Number (Numeric string is converted to a number).

  • MIN – Number, String, DateTime, LocalDateTime, LocalDate, LocalTime.

  • MAX – Number, String, DateTime, LocalDateTime, LocalDate, LocalTime.

    • When using a date for MIN/MAX functions, change “$DATE.date" to "$DATE" in the Aggregate field.

  • AVG – Number, DateTime, LocalDateTime, LocalDate, LocalTime (Numeric string is converted to a number).

  • CONCAT – Concatenates all records separated by a pipe (|).

  • UNIQUE_CONCAT – Concatenates all unique records separated by a pipe (|).

When you select the AVG function, the Snap rounds up all numeric values that have more than 16 digits. The AVG function handles the numeric values as below:

  • If the raw value is less than 16 with or without decimals, then the output value displays without any change.

    • Raw Value: 1234567890/1234567890.1234

    • Output Value: 1234567890/1234567890.1234

  • If the raw value is greater than or equal to 16 with or without decimals, then the output value is rounded off to 16 digits, including decimals.

    • Raw Value: 123456789012345.67

    • Output Value: 123456789012345.7

  • If the raw value is greater than 16 with or without decimals, then the output value is rounded up to 16 digits and expressed in exponential notation.

    • Raw Value: 1234567890123456789/1234567890123456789.123

    • Output Value: 1.234567890123457E+18

  • If the raw value is non-terminating decimal, then the output value is truncated to 16 digits.

    • Raw Value: 1.3333333333333333333333

    • Output Value: 1.333333333333333

Field*

Default value: [None]
Example:  $Total or DateTime.parse($service)

String/Expression

Specify a JSON path to the field on which the Aggregate function should be applied such as $items.item. Learn more: JSONPath - XPath for JSON.

 

Result field*

Default value:  [None]
Example:  Revenue

String

Specify the field name to be used for mapped data in the output. This value is the aggregate computed result corresponding to the GROUP-BY field values.

GROUP-BY fields*

 

 

 

 

 

 

 

Use this field set to define field paths and names. If you leave this field blank, the Snap produces only one output document.

Field

Default value: [None]
Example:  $.Product.Name

String/Expression

Specify a JSON path for the GROUP-BY field.

 

Output field

Default value:  [None]
Example:  ProductName

String

Specify the GROUP-BY field name to be used in the output map data. If left blank, the Field path is used instead.

Integer mode

Default value: Deselected
Example: Selected

 

Checkbox

Select this checkbox if you want the Snap to produce integer results rounded half up.

Sorted streams*

Default value: Unsorted
ExampleAscending

Dropdown list

Select an option to specify if the input documents are sorted or not.

This option enables the Snap to verify if the input is sorted as it processe