Common Words

On this Page

Overview

This Snap enables you to identify the most common words in the input dataset and compute the frequency with which they occur. It also offers you the ability to specify the number of most common words you want to include in the output. The input of this Snap should be an array of tokens, which can be generated by the Tokenizer Snap.

This Snap is required for the Bag of Words Snap. The output of this Snap should be connected to the second input view of the Bag of Words Snap

Expected Input and Output

  • Expected input: A document containing the field that contains the array of tokens.
  • Expected output: A document containing the most common words in the input dataset along with their frequency details.
  • Expected upstream Snaps: Any Snap that offers documents. For example: Tokenizer, or a combination of File Reader and JSON Parser.
  • Expected downstream Snaps: Any Snap that accepts documents. For example: Bag of Words, Mapper, or a combination of JSON Formatter and File Writer.

Prerequisites

None.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None.

Limitations and Known Issues

None.

Modes


Snap Settings


LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.
Token array field

Required. The field containing the array of tokens that must be examined for common words. This suggestible property is a JSONPath. Click the Suggestions icon to view a list of all available text fields. 

Default value: None.

Example: $text

Top words limit

Required. The number of most common words to be included in the output.

Default value: 100

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.


Examples


Computing the Frequency of Common Words in Tokenized Text Data

In this example, we shall demonstrate how you can use the Tokenizer and Common Words Snaps to compute the frequency of the most common words in a dataset.

Download this pipeline.

 Understanding the pipeline

This pipeline contains the following Snaps:

  1. File Reader: Picks up and reads the input file from SLFS.
  2. JSON Parser: Parses the JSON input and offers documents as output.
  3. Tokenizer: Converts sentences into an array of tokens.
  4. Common Words: Computes the frequency of the most common words in the input dataset.

The File Reader Snap reads an extract of the Yelp dataset (you can review the entire dataset here) and offers a binary stream as output. The JSON Parser Snap converts this binary stream into document stream, as shown below:


The field $text refers to sentences from Yelp user reviews. These sentences are used as input to the Tokenizer Snap using the following configurations:


As you can see, we selected $text for the Text field property. This is the content that will be tokenized and output as an array of tokens, as shown below:


As you can see in the screenshot above, each word in the input sentences has now become a token, and sentences in each input document become an array.

The Common Words Snap computes the frequency of each word that appears in the array of tokens. We need to configure the Common Words Snap to pick up data from the Tokenizer Snap and output frequency numbers related to the top 100 most common words, as shown below:


The pipeline, when run, offers the following output:


Based on the first 50 records of the Yelp dataset, during the pipeline validation, the most common words occur with the frequency shown above.

Download this pipeline.


Downloads

  File Modified

File Common_Words_Example.slp

Nov 27, 2018 by Rakesh Chaudhary

Snap Pack History

 Click to view/expand

4.27 (main12833)

  • No updates made.

4.26 (main11181)

  • No updates made.

4.25 (main9554)

  • No updates made.

4.24 (main8556)

  • No updates made.

4.23 (main7430)

  • No updates made.

4.22 (main6403)

  • No updates made.

4.21 (snapsmrc542)

  • No updates made.

4.20 Patch mlnlp8772

  • Removes the unused jcc-optional dependency from the ML Natural Language Processing Snap Pack.

4.20 (snapsmrc535)

  • No updates made.

4.19 (snapsmrc528)

  • No updates made.

4.18 (snapsmrc523)

  • No updates made.

4.17 Patch ALL7402

  • Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17 (snapsmrc515)

  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16 (snapsmrc508)

  • New Snap Pack. The Machine Learning Natural Language Processing Snap Pack contains Snaps that enable you to perform operations in natural language processing (NLP).

    This Snap Pack contains the following Snaps:

    • Tokenizer: Converts sentences into an array of tokens.
    • Common Words: Finds the most popular words in the dataset of input sentences.
    • Bag of Words: Vectorizes sentences into a set of numeric fields.