Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Overview


This Snap converts sentences into an array of tokens. A token, in the context of the ML Natural Language Processing Snap Pack, can be a word and / or special characters. In order to perform Natural Language Processing (NLP) operations with other Snaps in this Snap Pack, an array of tokens is required.

The Tokenizer Snap uses Apache OpenNLP Library.

Note

The Apache OpenNLP library is a machine-learning based toolkit for processing natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution.


Input and Output

  • Expected input: A document containing text fields.
  • Expected output: A document containing an array of tokens.
  • Expected upstream Snaps: Any Snap that offers documents. For example: Mapper, CSV Generator.
  • Expected downstream Snaps: Any Snap that accepts documents. For example: Common Words, Bag of Words, or Mapper.

Prerequisites

None.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views


Input

This Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.


Troubleshooting

None.

Limitations and Known Issues

None.

Modes

Snap Settings



Label

Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.

Text field

Required. The field containing the sentences that must be tokenized. This suggestible property is a JSONPath

Default value: None.

Example: $text

Word only

Select this property to not include spacial characters in the output. 

Default value: Selected.

Multiexcerpt include macro
nameSnap Execution
pageAnaplan Read


Multiexcerpt include macro
nameSnap_Execution_Introduced
pageAnaplan Read



Examples


Tokenizing Text Data

In this example, we show how to use the Tokenizer Snap to convert sentences into an array of tokens, which can be used in other NLP Snaps.

Download this pipeline.

Expand
titleUnderstanding the pipeline

This pipeline contains the following Snaps:

  1. CSV Generator: Generates a set of sentences as documents.
  2. Tokenizer: Converts sentences into an array of tokens.

The CSV Generator Snap outputs the following sentences:


These sentences are used as input to the Tokenizer Snap using the following configurations:


As you can see, we selected $text for the Text field property. This is the content that will be tokenized and output as an array of tokens.

The pipeline, when run, offers the following output:


As you can see, each word in the input sentences has now become a token, and sentences in each input document have become an array.

Download this pipeline.


Downloads

Attachments
patterns*.slp,*.zip

Excerpt Include
ML Natural Language Processing Snap Pack
ML Natural Language Processing Snap Pack
nopaneltrue