On this Page
This Snap converts sentences into an array of tokens. A token, in the context of the ML Natural Language Processing Snap Pack, can be a word and / or special characters. In order to perform Natural Language Processing (NLP) operations with other Snaps in this Snap Pack, an array of tokens is required.
The Tokenizer Snap uses Apache OpenNLP Library.
The Apache OpenNLP library is a machine-learning based toolkit for processing natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution. |
None.
Accounts are not used with this Snap.
Input | This Snap has exactly one document input view. |
---|---|
Output | This Snap has exactly one document output view. |
Error | This Snap has at most one document error view. |
None.
None.
Label | Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline. |
---|---|
Text field | Required. The field containing the sentences that must be tokenized. This suggestible property is a JSONPath. Default value: None. Example: $text |
Word only | Select this property to not include spacial characters in the output. Default value: Selected. |
In this example, we show how to use the Tokenizer Snap to convert sentences into an array of tokens, which can be used in other NLP Snaps.
Download this pipeline.
This pipeline contains the following Snaps:
The CSV Generator Snap outputs the following sentences: These sentences are used as input to the Tokenizer Snap using the following configurations: As you can see, we selected $text for the Text field property. This is the content that will be tokenized and output as an array of tokens. The pipeline, when run, offers the following output: As you can see, each word in the input sentences has now become a token, and sentences in each input document have become an array. Download this pipeline. |
Drag and drop pipeline examples (.slp, .zip) into this table outside of edit mode. Make sure the file name clearly indicates which example it is for or refer to it directly in the example's explanation.
Include a consolidated list of internal and external resources related to this Snap as a bulleted list. Also include any external URLs you may have added in the documentation above.