edu.stanford.nlp.pipeline
Class WhitespaceTokenizerAnnotator

java.lang.Object
  edu.stanford.nlp.pipeline.TokenizerAnnotator
      edu.stanford.nlp.pipeline.WhitespaceTokenizerAnnotator

All Implemented Interfaces:

Annotator

public class WhitespaceTokenizerAnnotator
extends TokenizerAnnotator

This annotator uses a WhitespaceTokenizer to split TextAnnotations into TokensAnnotations.
If either the property EOL_PROPERTY or the property NEWLINE_SPLITTER_PROPERTY defined in StanfordCoreNLP are present and set to true, newlines are returned as tokens. In practice, either will mean the newlines get removed by the sentence splitter.

Author:

John Bauer

Field Summary

static java.lang.String

EOL_PROPERTY

Constructor Summary

WhitespaceTokenizerAnnotator(java.util.Properties props)

Method Summary

Methods inherited from class edu.stanford.nlp.pipeline.TokenizerAnnotator

annotate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EOL_PROPERTY

public static final java.lang.String EOL_PROPERTY

See Also:

Constant Field Values

Constructor Detail

WhitespaceTokenizerAnnotator

public WhitespaceTokenizerAnnotator(java.util.Properties props)