On this Page
| Table of Contents | ||||
|---|---|---|---|---|
|
Overview
The Sample Snap is a Flow type Snap that enables you to generate a sample dataset from the input dataset. This sampling is carried out based on one of the following algorithms and with a predefined pass through percentage. The algorithms available are:
- Linear Split
- Streamable Sampling
- Strict Sampling
- Stratified Sampling
- Weighted Stratified Sampling
These algorithms are explained in the Snap Settings section below.
A random seed can also be provided to generate the same sample set for a given seed value. You can also optimize the Snap's usage of node memory by configuring the maximum memory in percentage that the Snap can use to buffer the input dataset. If the memory utilization is exceeded, the Snap writes the dataset into a temporary local file. This helps you avoid timeout errors when executing the pipeline.
Input and Output
Expected input: The input document from which the sample dataset is to be generated. The Snap accepts both numeric and categorical data; the stratified sampling and weighted stratified sampling algorithms require datasets containing categorical fields.
Expected output:
- First output: Document output containing the sample dataset.
- Second output: Document output containing the dataset that is not present in the first output.
Expected upstream Snaps: Snaps that provide a document output stream containing the dataset. For example, CSV Generator or a combination of File Reader and CSV Parser.
Expected downstream Snaps: Snaps that accept a document input. For example, Mapper or a combination of JSON Parser and File Writer.
Prerequisites
A basic understanding of the sampling algorithms supported by the Snap is preferable.
Configuring Accounts
Accounts are not used with this Snap.
Configuring Views
Input | This Snap has exactly one document input view. |
|---|---|
| Output | This Snap has at most two document output views. |
| Error | This Snap has at most one document error view. |
Troubleshooting
None
Limitations and Known Issues
None
Modes
- Ultra Pipelines: Works with Ultra Pipelines only when Streamable Sampling is selected as the sampling algorithm.
Snap Settings
| Label | Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Pass through percentage | Required. The number of records, as a percentage, that are to be passed through to the output. This value is treated differently based on the algorithm selected. Default value: 0.5
| ||||||||||||
| Algorithm | Required. The sampling algorithm to be used. Choose from one of the following options in the drop-down menu:
Default value: Streamable Sampling
| ||||||||||||
| Stratified field | Conditional. The field in the dataset containing classification information pertaining to the data. This is a suggestible property and lists all the fields in the incoming dataset. Select the field that is to be treated as the stratified field and the sampling is done based on this field. Example: Consider an employee record dataset containing fields such as Name, ID, Position, and Location. The fields Position, and Location help you classify the data, so the input in this property for this case is $Position or $Location. Default value: None | ||||||||||||
| Use random seed | If selected, Random seed is applied to the randomizer in order to get reproducible results. Default value: Selected | ||||||||||||
| Random seed | Conditional. This is required if the Use random seed property is selected. Number used as static seed for the randomizer. Default value: 12345
| ||||||||||||
| Maximum memory % | Required. The maximum portion of the node's memory, as a percentage, that can be utilized to buffer the incoming dataset. If this percentage is exceeded then the dataset is written to a temporary local file and then the sample generated from this temporary file. This configuration is useful in handling large datasets without over-utilization of the node memory. The minimum default memory to be used by the Snap is set at 100 MB. Default value: 10 | ||||||||||||
|
|
Multiexcerpt include macro name Temporary Files page Join
Example
Data Sampling
This example demonstrates all sampling algorithms applied to a document. Each of the following sampling algorithms is demonstrated:
- Streamable Sampling
- Strict Sampling
- Stratified Sampling
- Weighted Stratified Sampling
Download this pipeline.
| Expand | ||
|---|---|---|
| ||
The input is a CSV document generated by the CSV Generator Snap. A preview of the output from the CSV Generator is as shown below:
This document is passed to the Copy Snap where it generates five document streams, four of these go into the Sample Snap, and one goes into a Profile Snap. The Profile Snap generates a statistical profile of the incoming document, in this case the input document for the Sample Snaps. A preview of the output from the Profile Snap is as shown below:
There are two aspects of the input document based on the Profile Snap's output:
This data is useful in understanding how the Sample Snap creates a sample dataset for each sampling algorithm selected. Using the same pass-through percentage (50%), all four sampling algorithms are demonstrated here:
Download this pipeline. |
Downloads
| Multiexcerpt include macro | ||||
|---|---|---|---|---|
|
| Attachments | ||
|---|---|---|
|
| Insert excerpt | ||||||
|---|---|---|---|---|---|---|
|