edu.stanford.nlp.pipeline

Class WhitespaceTokenizerAnnotator

java.lang.Object

edu.stanford.nlp.pipeline.TokenizerAnnotator

edu.stanford.nlp.pipeline.WhitespaceTokenizerAnnotator

All Implemented Interfaces:

Annotator

public class WhitespaceTokenizerAnnotator
extends TokenizerAnnotator

This annotator uses a WhitespaceTokenizer to split TextAnnotations into TokensAnnotations.
If either the property EOL_PROPERTY or the property NEWLINE_SPLITTER_PROPERTY defined in StanfordCoreNLP are present and set to true, newlines are returned as tokens. In practice, either will mean the newlines get removed by the sentence splitter.

Author:

John Bauer

Nested Class Summary

Nested classes/interfaces inherited from interface edu.stanford.nlp.pipeline.Annotator

Annotator.Requirement

Field Summary

Fields

Modifier and Type	Field and Description
static String	EOL_PROPERTY

Fields inherited from interface edu.stanford.nlp.pipeline.Annotator

BINARIZED_TREES_REQUIREMENT, CLEAN_XML_REQUIREMENT, DETERMINISTIC_COREF_REQUIREMENT, GENDER_REQUIREMENT, GUTIME_REQUIREMENT, HEIDELTIME_REQUIREMENT, LEMMA_REQUIREMENT, NER_REQUIREMENT, NUMBER_REQUIREMENT, PARSE_AND_TAG, PARSE_REQUIREMENT, PARSE_TAG_BINARIZED_TREES, POS_REQUIREMENT, QUANTIFIABLE_ENTITY_NORMALIZATION_REQUIREMENT, RELATION_EXTRACTOR_REQUIREMENT, SSPLIT_REQUIREMENT, STANFORD_CLEAN_XML, STANFORD_DETERMINISTIC_COREF, STANFORD_GENDER, STANFORD_LEMMA, STANFORD_NER, STANFORD_PARSE, STANFORD_POS, STANFORD_REGEXNER, STANFORD_RELATION, STANFORD_SENTIMENT, STANFORD_SSPLIT, STANFORD_TOKENIZE, STANFORD_TRUECASE, STEM_REQUIREMENT, SUTIME_REQUIREMENT, TIME_WORDS_REQUIREMENT, TOKENIZE_AND_SSPLIT, TOKENIZE_REQUIREMENT, TOKENIZE_SSPLIT_NER, TOKENIZE_SSPLIT_PARSE, TOKENIZE_SSPLIT_PARSE_NER, TOKENIZE_SSPLIT_POS, TOKENIZE_SSPLIT_POS_LEMMA, TRUECASE_REQUIREMENT

Constructor Summary

Constructors

Constructor and Description
WhitespaceTokenizerAnnotator(Properties props)

Method Summary

Methods

Modifier and Type	Method and Description
Tokenizer<CoreLabel>	getTokenizer(Reader r) Abstract: returns a tokenizer

Methods inherited from class edu.stanford.nlp.pipeline.TokenizerAnnotator

annotate, requirementsSatisfied, requires

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EOL_PROPERTY

public static final String EOL_PROPERTY

See Also:

Constant Field Values

Constructor Detail

WhitespaceTokenizerAnnotator

public WhitespaceTokenizerAnnotator(Properties props)

Method Detail

getTokenizer

public Tokenizer<CoreLabel> getTokenizer(Reader r)

Description copied from class: TokenizerAnnotator

Abstract: returns a tokenizer

Specified by:

getTokenizer in class TokenizerAnnotator