On this Page

Snap type:

Transform

Description:

The Snap groups data from multiple input documents into each output document. Each output document contains an array of input data at the location specified by the Target field property. The size of an array is specified by the Group size property. The number of output documents is, the number of input documents divided by the Group size, rounded up, except when the Memory Sensitivity property is set to Dynamic, which allows the group size to vary dynamically.

Expected upstream Snaps: Any Snap with a document output view
Expected downstream Snaps: Any Snap with a document input view
Expected input: A document with a Map data
Expected output: A document with a list of input data as a value at the location specified by the Target field

Prerequisites:

All input documents should be of Map data type.

Support and limitations:

Does not work in Ultra Pipelines.

Account:

Accounts are not used with this Snap.

Views:

Input	This Snap has exactly one document input view.
Output	This Snap has exactly one document output view.
Error	This Snap has at most one document error view and produces zero or more documents in the view. The error view contains error, reason, resolution, and stack trace.

Settings

Label

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Target field

Required. Specifies the JSON path where the group array should be located within each output document.

Example: "grouped_data", "$group", "$group.list"

Default value: "group"

Memory Sensitivity

Required. Indicates the Snap's behavior towards memory changes. Choose one of the available options:

None: If selected, the size of every group will be the configured Group Size, except for the final group and partial groups created by the Flush Timeout feature
Dynamic: If selected, the size of each group will vary dynamically based on available memory, ranging from a maximum specified by Group Size and a minimum specified by Group Size.

This setting is disabled when the Group Size is set to 0.

Example: Dynamic
Default value: None

Group size

Required. Enter the number of input documents to be grouped into a single output document. A value of 0 instructs the Snap to group all the input documents into a single document. When Memory Sensitivity is Dynamic, this field specifies the maximum size of the group.

Minimum value: 0
Maximum value: No maximum value.

When the input stream ends, the Snap outputs the final group, regardless of the Group Size. For example, if the input stream has 105 documents in it, and the Group Size is 100, the Snap outputs one group of 100 and one group of 5.

This is an expression-enabled property; however, you can only pass data for this property using Pipeline parameters or expressions. This Snap does not support passing upstream data.

Example: 15000

Minimum value: 0

Maximum value: No maximum value.

Default value: 10

Min Group Size

Activated when Memory Sensitivity is set to Dynamic.

Enter the minimum number of input documents to be grouped into a single output document.

We recommend setting this value to 5% or less of the Group Size. It should not be higher than 10% of the Group Size. This setting is not applicable to the last group and flushed groups.

Flush Timeout

Required. Enter a non-zero value in this field to specify the number of seconds which can pass with no new input before the Snap should output a partial group, a group containing fewer than Group Size input documents.

When the Flush Timeout is 0, the Snap waits until it receives the messages specified in the Group Size field.

The Flush Timeout is useful in scenarios where the input stream never ends, or has long pauses as documents are read from it. In scenarios, where the Snap continually polls from an external system for new data, such as Kafka Consumer or Salesforce Subscriber Snaps, you can use the Flush Timeout field to specify a timeout so that the Snap always outputs whatever is available.

For example, if the Group Size is 100 and 105 records are currently available from Kafka application, the Snap passes output in two groups (100 and then 5), and continues to wait for more records. If the upstream Snap outputs another 15 records that are available, another group of 15 or more is passed, after the Flush Timeout is reached.

Example: 10

Default value: 0

Minimum memory (MB)

If the available memory is less than this property value while processing input documents, the Snap stops to fetch the next input document until more memory is available. This feature is disabled if this property value is 0.

Example: 500

Default value: 750

Out-of-memory timeout (minutes)

If the Snap pauses longer than this property value while waiting for more memory available, it throws an exception to prevent the system from running out of memory.

Example: 30

Default value: 20

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute

Examples

The input and output documents in code

Input and Output Documents in Code

Assume an input stream of five documents as follows:

[
    {
      "OrderNumber": 1,
      "OrderItem": "hamburger",
      "Stuff": [
        "apple",
        "pear",
        "peach"
      ]
    },
    {
      "OrderNumber": 1,
      "OrderItem": "fries",
      "Stuff": [
        "apple",
        "pear",
        "peach"
      ]
    },
    {
      "OrderNumber": 1,
      "OrderItem": "coke",
      "Stuff": [
        "apple",
        "pear",
        "peach"
      ]
    },
    {
      "OrderNumber": 2,
      "OrderItem": "hot dog",
      "Stuff": [
        "apple",
        "pear",
        "peach"
      ]
    },
    {
      "OrderNumber": 2,
      "OrderItem": "sprite",
      "Stuff": [
        "apple",
        "pear",
        "peach"
      ]
    }
  ]

If we set the Group size property to "2" and the Target field property to "$group.list", there will be three output documents as follows:

[
  {
    "group": {
      "list": [
        {
          "OrderNumber": 1,
          "OrderItem": "hamburger",
          "Stuff": [
            "apple",
            "pear",
            "peach"
          ]
        },
        {
          "OrderNumber": 1,
          "OrderItem": "fries",
          "Stuff": [
            "apple",
            "pear",
            "peach"
          ]
        }
      ]
    }
  },
  {
    "group": {
      "list": [
        {
          "OrderNumber": 1,
          "OrderItem": "coke",
          "Stuff": [
            "apple",
            "pear",
            "peach"
          ]
        },
        {
          "OrderNumber": 2,
          "OrderItem": "hot dog",
          "Stuff": [
            "apple",
            "pear",
            "peach"
          ]
        }
      ]
    }
  },
  {
    "group": {
      "list": [
        {
          "OrderNumber": 2,
          "OrderItem": "sprite",
          "Stuff": [
            "apple",
            "pear",
            "peach"
          ]
        }
      ]
    }
  }
]

The input and output documents in a pipeline

Input and Output Documents in a Pipeline

In this pipeline, the Group By N Snap groups the input documents into batches by the group size. The File Reader Snap passes the input documents to be parsed and sorted by the Group By N Snap.

The File Reader Snap passes the input document order.json to be grouped into batches by size.

The JSON Parser Snap parses the binary input from the File Reader Snap:

The Sort Snap sorts the input documents in ascending order. The respective output preview:

The Group By N Snap groups the Target Field $group.list in the size of 2 meaning each group will have a batch of two output documents.

The output preview from the Group By N Snap with the grouping into batch of output documents by Group size 2: