Bag of Words

On this Page

Overview


This Snap vectorizes sentences into a set of numeric fields. It takes two inputs:

  1. An array of tokenized words drawn from a set of input sentences. If your sentences are not already tokenized, use the Tokenizer Snap.
  2. A document containing the most common words in the training set. You can generate this document using the Common Words Snap.

It then processes the data received and outputs a count frequency of the most common words in the text in each sentence.

For example: in the first output, say we have an array containing all the tokens in each sentence in a dataset; and the frequency of the 100 most common words in the second input. The Bag of Words Snap enables you to list out the number of times each of the 100 most common words appear in each input sentence.

Expected Input and Output

Expected input:

  • First input view: An array of tokenized words drawn from a set of input sentences. If your sentences are not already tokenized, use the Tokenizer Snap.
  • Second input view: A document containing the most common words in a given dataset, together with their frequency. You can generate this document using the Common Words Snap.

Expected output: A document containing the count frequency of the most common words in the text field.

Expected upstream Snaps:

  • First input view: Any Snap that offers documents. For example: Tokenizer or Mapper.
  • Second input view: Any Snap that offers documents. For example: Common Words, or a combination of File Reader and JSON Parser Snaps.

Expected downstream Snaps: Any Snap that accepts documents. For example: Cross Validator, Trainer, or a combination of JSON Formatter and File Writer Snaps.

Prerequisites

None.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has exactly two document input views.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None.

Limitations and Known Issues

None.

Modes


Snap Settings


LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.
Token array field

Required. The field containing the array of tokens on which you want to apply the Bag of Words technique. This suggestible property is a JSONPath

Default value: None

Example: $text

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:
  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.


Examples


Identifying the Occurrence Frequency of the Top-100 Common Words in a Tokenized Input Dataset

In this example, we demonstrate how you can use the Bag of Words Snap to capture the frequency with which members of a set of common words appear in a dataset of tokenized sentences.


Download this pipeline.

 Understanding the pipeline

This pipeline contains the following Snaps:

  1. JSON Generator: Contains an array of tokens created using the Yelp dataset. For details on how you can create such an array, see the example in the Tokenizer documentation.
  2. Mapper: Enables you to retain only the text part of the input document, indicated in the Snap Settings as $text.
  3. File Reader: Picks up and reads the contents of the yelp_common_words.json file, which contains the top 100 common words in the Yelp dataset. For details on how you can create this file, review the example in the Tokenizer documentation.
  4. JSON Parser: Parses the input JSON data and offers documents as input to the Bag of Words Snap.
  5. Bag of Words: Generates a document detailing the frequency with which members of the set of common words received from the File Reader and JSON Parser Snaps appear in the dataset of tokenized sentences received from the JSON Generator and Mapper Snaps.

The JSON Generator Snap contains an array of tokens created using an extract of the Yelp dataset (you can review the entire dataset here) and makes it available to the Mapper Snap as a document. If you do not have an array of tokens, you can create one using the Tokenizer Snap.

 


The Mapper Snap picks up the tokenized data in the input file coming from the JSON Generator Snap and maps it to the field $text, making all the relevant data accessible to the Bag of Words Snap:


The Bag of Words Snap is configured to pick up the input data coming in from the Mapper Snap:


The Bag of Words Snap runs the Mapper output (the array of tokenized words in each sentence) against the JSON Parser output (the array listing out the frequency of the 100 most common words in the same dataset) and creates a document detailing the frequency with which the common words appear in each sentence.

Download this pipeline.

Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide Pipeline parameters as applicable.

  File Modified

File BagOfWords_Example.slp

Nov 27, 2018 by Rakesh Chaudhary

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721Nov 8, 2023StableUpdated and certified against the current SnapLogic Platform release.

August 2023

main22460

Aug 16, 2023

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023

main21015 

Stable

Upgraded with the latest SnapLogic Platform release.

February 2023main19844
 
StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944
 
Stable

Upgraded with the latest SnapLogic Platform release.

August 2022main17386
 
Stable

Upgraded with the latest SnapLogic Platform release.

4.29main15993
 
StableUpgraded with the latest SnapLogic Platform release.
4.28main14627
 
Stable

Upgraded with the latest SnapLogic Platform release.

4.27main12833
 
StableUpgraded with the latest SnapLogic Platform release.
4.26main11181
 
StableUpgraded with the latest SnapLogic Platform release.
4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
 
StableUpgraded with the latest SnapLogic Platform release.
4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542
 
StableUpgraded with the latest SnapLogic Platform release.
4.20 Patchmlnlp8772
 
Latest

Removes the unused jcc-optional dependency from the ML Natural Language Processing Snap Pack.

4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19snapsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18snapsmrc523
 
StableUpgraded with the latest SnapLogic Platform release.
4.17 PatchALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508
 
Stable
  • New Snap Pack. The Machine Learning Natural Language Processing Snap Pack contains Snaps that enable you to perform operations in natural language processing (NLP).

    This Snap Pack contains the following Snaps:

    • Tokenizer: Converts sentences into an array of tokens.
    • Common Words: Finds the most popular words in the dataset of input sentences.
    • Bag of Words: Vectorizes sentences into a set of numeric fields.