## edu.stanford.nlp.process
Class TokenizerAdapter

    java.lang.Object
      edu.stanford.nlp.process.AbstractTokenizer<String>
          edu.stanford.nlp.process.TokenizerAdapter

All Implemented Interfaces:
  Tokenizer<String>, Iterator<String>

Direct Known Subclasses:
  PennTreebankTokenizer

    public class TokenizerAdapter

extends AbstractTokenizer<String>

This class adapts between a `java.io.StreamTokenizer` and a `edu.stanford.nlp.process.Tokenizer`.

Author:
  Christopher Manning

Field Summary
`protected String` `eolString`
`protected StreamTokenizer` `st`

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
`nextToken`

Constructor Summary
`TokenizerAdapter(StreamTokenizer st)` Create a new `TokenizerAdaptor`.

Method Summary
`String` `getNext()` Internally fetches the next token.
`boolean` `isEol(String str)` Say whether the `String` is the end-of-line token for this tokenizer.
`void` `setEolString(String eolString)` Set the `String` returned when the inner tokenizer returns an end-of-line token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
`hasNext, next, peek, remove, tokenize`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### st

    protected StreamTokenizer st

### eolString

    protected String eolString

Constructor Detail

### TokenizerAdapter

    public TokenizerAdapter(StreamTokenizer st)

  Create a new `TokenizerAdaptor`. In general, it is recommended that the passed in `StreamTokenizer` should have had `resetSyntax()` done to it, so that numbers are returned as entered as tokens of type `String`, though this code will cope as best it can.

  Parameters:
    `st` - The internal `java.io.StreamTokenizer`

Method Detail

### getNext

    public String getNext()

  Internally fetches the next token.

  Specified by:
    `getNext` in class `AbstractTokenizer<String>`
  Returns:
    the next token in the token stream, or null if none exists.

### setEolString

    public void setEolString(String eolString)

  Set the `String` returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to `eolIsSignificant(true)`.

  Parameters:
    `eolString` - The String used to represent eol. It is not allowed to be `null` (which would confuse line ends and file end)

### isEol

    public boolean isEol(String str)

  Say whether the `String` is the end-of-line token for this tokenizer.

  Parameters:
    `str` - The String being tested
  Returns:
    Whether it is the end-of-line token

Stanford NLP Group