Tokenizer

This page is no longer maintained (). For the most current information, go to Tokenizer.

On this Page

Overview


This Snap converts sentences into an array of tokens. A token, in the context of the ML Natural Language Processing Snap Pack, can be a word and / or special characters. In order to perform Natural Language Processing (NLP) operations with other Snaps in this Snap Pack, an array of tokens is required.

The Tokenizer Snap uses Apache OpenNLP Library.

The Apache OpenNLP library is a machine-learning based toolkit for processing natural language text. It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution.

Input and Output

  • Expected input: A document containing text fields.
  • Expected output: A document containing an array of tokens.
  • Expected upstream Snaps: Any Snap that offers documents. For example: Mapper, CSV Generator.
  • Expected downstream Snaps: Any Snap that accepts documents. For example: Common Words, Bag of Words, or Mapper.

Prerequisites

None.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None.

Limitations and Known Issues

None.

Modes

Snap Settings


Label

Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.

Text field

Required. The field containing the sentences that must be tokenized. This suggestible property is a JSONPath

Default value: None.

Example: $text

Word only

Select this property to not include spacial characters in the output. 

Default value: Selected.

Snap Execution

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default ValueExecute only
Example: Validate & Execute


Examples


Tokenizing Text Data

In this example, we show how to use the Tokenizer Snap to convert sentences into an array of tokens, which can be used in other NLP Snaps.

Download this pipeline.

 Understanding the pipeline

This pipeline contains the following Snaps:

  1. CSV Generator: Generates a set of sentences as documents.
  2. Tokenizer: Converts sentences into an array of tokens.

The CSV Generator Snap outputs the following sentences:


These sentences are used as input to the Tokenizer Snap using the following configurations:


As you can see, we selected $text for the Text field property. This is the content that will be tokenized and output as an array of tokens.

The pipeline, when run, offers the following output:


As you can see, each word in the input sentences has now become a token, and sentences in each input document have become an array.

Download this pipeline.

Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the pipeline into the SnapLogic application.
  2. Configure Snap accounts as applicable.
  3. Provide pipeline parameters as applicable.

  File Modified

File Tokenizer_Example.slp

Nov 27, 2018 by Rakesh Chaudhary

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721Nov 8, 2023StableUpdated and certified against the current SnapLogic Platform release.

August 2023

main22460

Aug 16, 2023

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023

main21015 

Stable

Upgraded with the latest SnapLogic Platform release.

February 2023main19844
 
StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944
 
Stable

Upgraded with the latest SnapLogic Platform release.

August 2022main17386
 
Stable

Upgraded with the latest SnapLogic Platform release.

4.29main15993
 
StableUpgraded with the latest SnapLogic Platform release.
4.28main14627
 
Stable

Upgraded with the latest SnapLogic Platform release.

4.27main12833
 
StableUpgraded with the latest SnapLogic Platform release.
4.26main11181
 
StableUpgraded with the latest SnapLogic Platform release.
4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
 
StableUpgraded with the latest SnapLogic Platform release.
4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542
 
StableUpgraded with the latest SnapLogic Platform release.
4.20 Patchmlnlp8772
 
Latest

Removes the unused jcc-optional dependency from the ML Natural Language Processing Snap Pack.

4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19snapsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18snapsmrc523
 
StableUpgraded with the latest SnapLogic Platform release.
4.17 PatchALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16snapsmrc508
 
Stable
  • New Snap Pack. The Machine Learning Natural Language Processing Snap Pack contains Snaps that enable you to perform operations in natural language processing (NLP).

    This Snap Pack contains the following Snaps:

    • Tokenizer: Converts sentences into an array of tokens.
    • Common Words: Finds the most popular words in the dataset of input sentences.
    • Bag of Words: Vectorizes sentences into a set of numeric fields.