This pipeline contains the following Snaps:
- File Reader: Picks up and reads the input file from SLFS.
- JSON Parser: Parses the JSON input and offers documents as output.
- Tokenizer: Converts sentences into an array of tokens.
- Common Words: Computes the frequency of the most common words in the input dataset.
The File Reader Snap reads an extract of the Yelp dataset (you can review the entire dataset here) and offers a binary stream as output. The JSON Parser Snap converts this binary stream into document stream, as shown below:
The field $text refers to sentences from Yelp user reviews. These sentences are used as input to the Tokenizer Snap using the following configurations:
As you can see, we selected $text for the Text field property. This is the content that will be tokenized and output as an array of tokens, as shown below:
As you can see in the screenshot above, each word in the input sentences has now become a token, and sentences in each input document become an array.
The Common Words Snap computes the frequency of each word that appears in the array of tokens. We need to configure the Common Words Snap to pick up data from the Tokenizer Snap and output frequency numbers related to the top 100 most common words, as shown below:
The pipeline, when run, offers the following output:
Based on the first 50 records of the Yelp dataset, during the pipeline validation, the most common words occur with the frequency shown above.
Download this pipeline.