edu.stanford.nlp.trees

Class PennTreebankTokenizer

java.lang.Object

edu.stanford.nlp.process.AbstractTokenizer<String>

edu.stanford.nlp.process.TokenizerAdapter

edu.stanford.nlp.trees.PennTreebankTokenizer

All Implemented Interfaces:

Tokenizer<String>, Iterator<String>

Direct Known Subclasses:

ArabicTreebankTokenizer

public class PennTreebankTokenizer
extends TokenizerAdapter

Builds a tokenizer for English PennTreebank (release 2) trees. This is currently internally implemented via a java.io.StreamTokenizer.

Author:

Christopher Manning, Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.TokenizerAdapter

eolString, st

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

Constructors

Constructor and Description
PennTreebankTokenizer(Reader r)

Method Summary

Methods inherited from class edu.stanford.nlp.process.TokenizerAdapter

getNext, isEol, setEolString

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PennTreebankTokenizer

public PennTreebankTokenizer(Reader r)