## edu.stanford.nlp.process
Interface Tokenizer<T>

All Superinterfaces:
  Iterator<T>

All Known Implementing Classes:
  AbstractTokenizer, CHTBTokenizer, PennTreebankTokenizer, PTBTokenizer, TokenizerAdapter, WhitespaceTokenizer, XMLBeginEndIterator

    public interface Tokenizer<T>

extends Iterator<T>

Tokenizers break up text into individual Objects. These objects may be Strings, Words, or other Objects. A Tokenizer extends the Iterator interface, but provides a lookahead operation `peek()`. An implementation of this interface is expected to have a constructor that takes a single argument, a Reader.

Author:
  Teg Grenager (grenager@stanford.edu)

Method Summary
`boolean` `hasNext()` Returns `true` if and only if this Tokenizer has more elements.
`T` `next()` Returns the next token from this Tokenizer.
`T` `peek()` Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().
`void` `remove()` Removes from the underlying collection the last element returned by the iterator.
`List<T>` `tokenize()` Returns all tokens of this Tokenizer as a List for convenience.

Method Detail

### next

    T next()

  Returns the next token from this Tokenizer.

  Specified by:
    `next` in interface `Iterator<T>`
  Returns:
    the next token in the token stream.
  Throws:
    `NoSuchElementException` - if the token stream has no more tokens.

### hasNext

    boolean hasNext()

  Returns `true` if and only if this Tokenizer has more elements.

  Specified by:
    `hasNext` in interface `Iterator<T>`

### remove

    void remove()

  Removes from the underlying collection the last element returned by the iterator. This is an optional operation for Iterators - a Tokenizer normally would not support it. This method can be called only once per call to next.

  Specified by:
    `remove` in interface `Iterator<T>`

### peek

    T peek()

  Returns the next token, without removing it, from the Tokenizer, so that the same token will be again returned on the next call to next() or peek().

  Returns:
    the next token in the token stream.
  Throws:
    `NoSuchElementException` - if the token stream has no more tokens.

### tokenize

    List<T> tokenize()

  Returns all tokens of this Tokenizer as a List for convenience.

Stanford NLP Group