On this Page
This Snap vectorizes sentences into a set of numeric fields. It takes two inputs:
A document containing the most common words in the training set. You can generate this document using the Common Words Snap.
It then processes the data received and outputs a count frequency of the most common words in the text in each sentence.
For example: in the first output, say we have an array containing all the tokens in each sentence in a dataset; and the frequency of the 100 most common words in the second input. The Bag of Words Snap enables you to list out the number of times each of the 100 most common words appear in each input sentence.
Expected input:
Expected output: A document containing the count frequency of the most common words in the text field.
Expected upstream Snaps:
Expected downstream Snaps: Any Snap that accepts documents. For example: Cross Validator, Trainer, or a combination of JSON Formatter and File Writer Snaps.
None.
Accounts are not used with this Snap.
Input | This Snap has exactly two document input views. |
---|---|
Output | This Snap has exactly one document output view. |
Error | This Snap has at most one document error view. |
None.
None.
Label | Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline. |
---|---|
Token array field | Required. The field containing the array of tokens on which you want to apply the Bag of Words technique. This suggestible property is a JSONPath. Default value: None Example: $text |
In this example, we demonstrate how you can use the Bag of Words Snap to capture the frequency with which members of a set of common words appear in a dataset of tokenized sentences.
Download this pipeline.
This pipeline contains the following Snaps:
The JSON Generator Snap contains an array of tokens created using an extract of the Yelp dataset (you can review the entire dataset here) and makes it available to the Mapper Snap as a document. If you do not have an array of tokens, you can create one using the Tokenizer Snap.
The Mapper Snap picks up the tokenized data in the input file coming from the JSON Generator Snap and maps it to the field $text, making all the relevant data accessible to the Bag of Words Snap: The Bag of Words Snap is configured to pick up the input data coming in from the Mapper Snap: The Bag of Words Snap runs the Mapper output (the array of tokenized words in each sentence) against the JSON Parser output (the array listing out the frequency of the 100 most common words in the same dataset) and creates a document detailing the frequency with which the common words appear in each sentence. Download this pipeline. |
Drag and drop pipeline examples (.slp, .zip) into this table outside of edit mode. Make sure the file name clearly indicates which example it is for or refer to it directly in the example's explanation.