Group By N
On this Page
Snap type: | Transform | |||||||
---|---|---|---|---|---|---|---|---|
Description: | The Snap groups data from multiple input documents into each output document. Each output document contains an array of input data at the location specified by the Target field property. The size of an array is specified by the Group size property. The number of output documents is, the number of input documents divided by the Group size, rounded up, except when the Memory Sensitivity property is set to Dynamic, which allows the group size to vary dynamically.
| |||||||
Prerequisites: | All input documents should be of Map data type. | |||||||
Support and limitations: | Does not work in Ultra Pipelines. | |||||||
Account: | Accounts are not used with this Snap. | |||||||
Views: |
| |||||||
Settings | ||||||||
Label | Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | |||||||
Target field | Required. Specifies the JSON path where the group array should be located within each output document. Example: "grouped_data", "$group", "$group.list" Default value: "group" | |||||||
Memory Sensitivity | Required. Indicates the Snap's behavior towards memory changes. Choose one of the available options:
| |||||||
Group size | Required. Enter the number of input documents to be grouped into a single output document. A value of 0 instructs the Snap to group all the input documents into a single document. When Memory Sensitivity is Dynamic, this field specifies the maximum size of the group.
When the input stream ends, the Snap outputs the final group, regardless of the Group Size. For example, if the input stream has 105 documents in it, and the Group Size is 100, the Snap outputs one group of 100 and one group of 5. This is an expression-enabled property; however, you can only pass data for this property using Pipeline parameters or expressions. This Snap does not support passing upstream data. Example: 15000 Minimum value: 0 Maximum value: No maximum value. Default value: 10 | |||||||
Min Group Size | Activated when Memory Sensitivity is set to Dynamic. Enter the minimum number of input documents to be grouped into a single output document. We recommend setting this value to 5% or less of the Group Size. It should not be higher than 10% of the Group Size. This setting is not applicable to the last group and flushed groups. | |||||||
Flush Timeout | Required. Enter a non-zero value in this field to specify the number of seconds which can pass with no new input before the Snap should output a partial group, a group containing fewer than Group Size input documents. When the Flush Timeout is 0, the Snap waits until it receives the messages specified in the Group Size field. The Flush Timeout is useful in scenarios where the input stream never ends, or has long pauses as documents are read from it. In scenarios, where the Snap continually polls from an external system for new data, such as Kafka Consumer or Salesforce Subscriber Snaps, you can use the Flush Timeout field to specify a timeout so that the Snap always outputs whatever is available. For example, if the Group Size is 100 and 105 records are currently available from Kafka application, the Snap passes output in two groups (100 and then 5), and continues to wait for more records. If the upstream Snap outputs another 15 records that are available, another group of 15 or more is passed, after the Flush Timeout is reached. Example: 10 Default value: 0 | |||||||
Minimum memory (MB) | If the available memory is less than this property value while processing input documents, the Snap stops to fetch the next input document until more memory is available. This feature is disabled if this property value is 0. Example: 500 Default value: 750 | |||||||
Out-of-memory timeout (minutes) | If the Snap pauses longer than this property value while waiting for more memory available, it throws an exception to prevent the system from running out of memory. Example: 30 Default value: 20 | |||||||
Snap Execution | Select one of the three modes in which the Snap executes. Available options are:
Default Value: Execute only |