edu.stanford.nlp.process

Class LexerTokenizer

java.lang.Object

edu.stanford.nlp.process.AbstractTokenizer<String>

edu.stanford.nlp.process.LexerTokenizer

All Implemented Interfaces:

Tokenizer<String>, Iterator<String>

Direct Known Subclasses:

NegraPennTokenizer

public class LexerTokenizer
extends AbstractTokenizer<String>

An implementation of Tokenizer designed to work with Lexer implementing classes. Throw in a Lexer on construction and you get a Tokenizer.

Author:

Roger Levy

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

Constructors

Constructor and Description
LexerTokenizer(Lexer l)
LexerTokenizer(Lexer l, Reader r)

Method Summary

Methods

Modifier and Type	Method and Description
protected String	getNext() Internally fetches the next token.
static void	main(String[] args) for testing only

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LexerTokenizer

public LexerTokenizer(Lexer l)

LexerTokenizer

public LexerTokenizer(Lexer l,
              Reader r)

Method Detail

getNext

protected String getNext()

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<String>

Returns:

the next token in the token stream, or null if none exists.

main

public static void main(String[] args)
                 throws IOException

for testing only

Throws:

IOException