/
Group By Fields

Group By Fields

Owned by Kalpana Malladi
Last updated: Jun 05, 2024 by Kalpana Malladi

In this article

Snap type:

Transform

Description:

The Snap groups input documents by the field values into batches of output documents. Each batch is an output document with a list of input Map data as a value at the location specified by the Target field property. Input documents with the same group-by field values are grouped into the same output document.


The Snap expects the input documents with the same group-by field values to be contiguous and whenever the group-by field values change, the Snap produces a new output document. Therefore, if all input documents with the same group-by field values are expected to be grouped into one output document, the Sort Snap can be used in front of the Group By Fields Snap so that the input document stream are sorted by the group-by field values.


  • Expected upstream Snaps: Any Snap with a document output view
  • Expected downstream Snaps: Any Snap with a document input view
  • Expected input: A document with Map data
  • Expected output: A document with a list of input Map data as a value at the location specified by the Target field
Prerequisites:

All input documents should be of Map data type and contain values specified by the Fields property.

Support and limitations:Does not work in Ultra Pipelines.
Account: 

Accounts are not used with this Snap.

Views:
InputThis Snap has exactly one document input view.
OutputThis Snap has exactly one document output view. The Snap is configured with a second output view to get statistics of the input data
ErrorThis Snap has at most one document error view and produces zero or more documents in the view. The error view contains error, reason, resolution and stack trace.

Settings

Label


Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Fields



Required. The fields to group by.

Example: $OrderNumber

Default value: [None]

Memory Sensitivity

Required. Indicates the Snap's behavior towards memory changes. Choose one of the available options:

  • None: If selected, it groups input documents by the field values into batches of output documents.

  • Dynamic: If selected, groups may be split into multiple parts, depending on memory availability. The group size to scale against each group is determined statistically from the groups already processed (mean group size + one standard deviation)

Default value: [None]

Min. Part Size

Activated when Memory Sensitivity is set to Dynamic.

Enter the minimum part size that you want Snap to split larger groups into multiple parts. 

This limit does not apply to the last part of the multi-part group or a single part of the group that's smaller than the size of the part mentioned here. 

Example: 100

Default value: 10

Target field


Required. Target field name to be used as a key in the output document or a JSON path where a list of input Map data would be located.

Example: batch

Default value: group

Minimum memory (MB)

If the available memory is less than this property value while processing input documents, the Snap stops to fetch the next input document until more memory is available. This feature is disabled if this property value is 0.

Example: 500

Default value: 750

Out-of-memory timeout (minutes)

If the Snap pauses longer than this property value while waiting for more memory available, it throws an exception to prevent the system from running out of memory.


Example: 30

Default value: 20

Snap Execution



Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute

Examples

 Input and output documents batched by the group name and fields

Input and Output Documents Batched by the Group Name and Fields

In this pipeline, the Group By Fields Snap groups the input documents into a batch of output documents with the same group by the field property.

The JSON Generator Snap passes the values to be batched into groups by fields.


The Sort Snap Sorts the input documents into ascending order, the respective output preview:


The Group By Fields Snap groups the documents by group name and fields.

The output preview from the Group By Fields Snap grouped by the order number and the group fields:


The output preview in the table format:


 Input and output documents by code

Input and Output Documents by Code

Assume an input stream of nine documents as follows:

[
    {
      "OrderNumber": 1,
      "something": "abc"
    },
    {
      "OrderNumber": 1,
      "something": "def"
    },
    {
      "OrderNumber": 1,
      "something": "ghi"
    },
    {
      "OrderNumber": 2,
      "something": "jkl"
    },
    {
      "OrderNumber": 2,
      "something": "mno"
    },
    {
      "OrderNumber": 2,
      "something": "pqr"
    },
    {
      "OrderNumber": 3,
      "something": "stu"
    },
    {
      "OrderNumber": 3,
      "something": "vwx"
    },
    {
      "OrderNumber": 4,
      "something": "yz"
    }
]


 If we set the Fields property to "$OrderNumber" and the Target field property to "$group.items", there will be four output documents as follows:


[
  {
    "groupBy": {
      "OrderNumber": 1
    },
    "group": {
      "items": [
        {
          "OrderNumber": 1,
          "something": "abc"
        },
        {
          "OrderNumber": 1,
          "something": "def"
        },
        {
          "OrderNumber": 1,
          "something": "ghi"
        }
      ]
    }
  },
  {
    "groupBy": {
      "OrderNumber": 2
    },
    "group": {
      "items": [
        {
          "OrderNumber": 2,
          "something": "jkl"
        },
        {
          "OrderNumber": 2,
          "something": "mno"
        },
        {
          "OrderNumber": 2,
          "something": "pqr"
        }
      ]
    }
  },
  {
    "groupBy": {
      "OrderNumber": 3
    },
    "group": {
      "items": [
        {
          "OrderNumber": 3,
          "something": "stu"
        },
        {
          "OrderNumber": 3,
          "something": "vwx"
        }
      ]
    }
  },
  {
    "groupBy": {
      "OrderNumber": 4
    },
    "group": {
      "items": [
        {
          "OrderNumber": 4,
          "something": "yz"
        }
      ]
    }
  }
]

Related Content