Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## edu.stanford.nlp.process
Class PTBTokenizer<T extends HasWord>

    java.lang.Object
      edu.stanford.nlp.process.AbstractTokenizer<T>
          edu.stanford.nlp.process.PTBTokenizer<T>

All Implemented Interfaces:
  Tokenizer<T>, Iterator<T>

    public class PTBTokenizer<T extends HasWord>

extends AbstractTokenizer<T>

Tokenizer implementation that conforms to the Penn Treebank tokenization conventions. This tokenizer is a Java implementation of Professor Chris Manning's Flex tokenizer, pgtt-treebank.l. It reads raw text and outputs tokens as edu.stanford.nlp.trees.Words in the Penn treebank format. It can optionally return carriage returns as tokens.

Author:
  Tim Grow, Teg Grenager (grenager@stanford.edu), Christopher Manning, Jenny Finkel (integrating in invertible PTB tokenizer)

Nested Class Summary
`static class` | `PTBTokenizer.PTBTokenizerFactory<T extends HasWord>`

Field Summary

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
`nextToken`

Constructor Summary
`PTBTokenizer(Reader r, LexedTokenFactory<T> tokenFactory, String options)` Constructs a new PTBTokenizer with a custom LexedTokenFactory.

Method Summary
`static TokenizerFactory<Word>` | `factory()`
`static TokenizerFactory<CoreLabel>` | `factory(boolean tokenizeNLs, boolean invertible)`
`static
`<T extends HasWord> TokenizerFactory<T>`
` | `factory(boolean tokenizeNLs, LexedTokenFactory<T> factory)`
`static
`<T extends HasWord> TokenizerFactory<T>`
` | `factory(LexedTokenFactory<T> factory, String options)`
`protected T` | `getNext()` Internally fetches the next token.
`static String` | `labelList2Text(List<? extends HasWord> ptbWords)` Returns a presentable version of the given PTB-tokenized words.
`static void` | `main(String[] args)` Reads files named as arguments and print their tokens, by default as one per line.
`static PTBTokenizer<Word>` | `newPTBTokenizer(Reader r)` Constructs a new PTBTokenizer that returns Word tokens and which treats carriage returns as normal whitespace.
`static PTBTokenizer<Word>` | `newPTBTokenizer(Reader r, boolean tokenizeNLs)` Constructs a new PTBTokenizer that optionally returns newlines as their own token.
`static PTBTokenizer<CoreLabel>` | `newPTBTokenizer(Reader r, boolean tokenizeNLs, boolean invertible)` Constructs a new PTBTokenizer that makes CoreLabel tokens.
`static String` | `ptb2Text(List<String> ptbWords)` Returns a presentable version of the given PTB-tokenized words.
`static int` | `ptb2Text(Reader ptbText, Writer w)` Writes a presentable version of the given PTB-tokenized text.
`static String` | `ptb2Text(String ptbText)` Returns a presentable version of the given PTB-tokenized text.
`static String` | `ptbToken2Text(String ptbText)` Returns a presentable version of a given PTB token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
`hasNext, next, peek, remove, tokenize`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

### PTBTokenizer

    public PTBTokenizer(Reader r,
                        LexedTokenFactory<T> tokenFactory,
                        String options)

  Constructs a new PTBTokenizer with a custom LexedTokenFactory. Many options for tokenization and what is returned can be set via the options String. This is the new recommended constructor!

  Parameters:
    `r` - The Reader to read tokens from
    `tokenFactory` - The LexedTokenFactory to use to create tokens from the text.
    `options` - Options to the lexer. See the extensive documentation in PTBLexer. The String may be null or empty, which means that all traditional PTB normalizations are done. You can pass in "ptb3Escaping=false" and have no normalizations done (that is, the behavior of the old suppressEscaping=true option).

Method Detail

### newPTBTokenizer

    public static PTBTokenizer<Word> newPTBTokenizer(Reader r)

  Constructs a new PTBTokenizer that returns Word tokens and which treats carriage returns as normal whitespace.

  Parameters:
    `r` - The Reader whose contents will be tokenized
  Returns:
    A PTBTokenizer that tokenizes a stream to objects of type `Word`

### newPTBTokenizer

    public static PTBTokenizer<Word> newPTBTokenizer(Reader r,
                                                     boolean tokenizeNLs)

  Constructs a new PTBTokenizer that optionally returns newlines as their own token. NLs come back as Words whose text is the value of `PTBLexer.NEWLINE_TOKEN`.

  Parameters:
    `r` - The Reader to read tokens from
    `tokenizeNLs` - Whether to return newlines as separate tokens (otherwise they normally disappear as whitespace)
  Returns:
    A PTBTokenizer which returns Word tokens

### newPTBTokenizer

    public static PTBTokenizer<CoreLabel> newPTBTokenizer(Reader r,
                                                          boolean tokenizeNLs,
                                                          boolean invertible)

  Constructs a new PTBTokenizer that makes CoreLabel tokens. It optionally returns carriage returns as their own token. CRs come back as Words whose text is the value of `PTBLexer.NEWLINE_TOKEN`.

  Parameters:
    `r` - The Reader to read tokens from
    `tokenizeNLs` - Whether to return newlines as separate tokens (otherwise they normally disappear as whitespace)
    `invertible` - if set to true, then will produce CoreLabels which will have fields for the string before and after, and the character offsets
  Returns:
    A PTBTokenizer which returns CoreLabel objects

### getNext

    protected T getNext()

  Internally fetches the next token.

  Specified by:
    `getNext` in class `AbstractTokenizer<T extends HasWord>`
  Returns:
    the next token in the token stream, or null if none exists.

### ptb2Text

    public static String ptb2Text(String ptbText)

  Returns a presentable version of the given PTB-tokenized text. PTB tokenization splits up punctuation and does various other things that makes simply joining the tokens with spaces look bad. So join the tokens with space and run it through this method to produce nice looking text. It's not perfect, but it works pretty well.

  Parameters:
    `ptbText` - A String in PTB3-escaped form
  Returns:
    An approximation to the original String

### ptbToken2Text

    public static String ptbToken2Text(String ptbText)

  Returns a presentable version of a given PTB token. For instance, it transforms -PTB- into (.

### ptb2Text

    public static int ptb2Text(Reader ptbText,
                               Writer w)
                        throws IOException

  Writes a presentable version of the given PTB-tokenized text. PTB tokenization splits up punctuation and does various other things that makes simply joining the tokens with spaces look bad. So join the tokens with space and run it through this method to produce nice looking text. It's not perfect, but it works pretty well.

  Throws:
    `IOException`

### ptb2Text

    public static String ptb2Text(List<String> ptbWords)

  Returns a presentable version of the given PTB-tokenized words. Pass in a List of Strings and this method will join the words with spaces and call `ptb2Text(String)` on the output.

  Parameters:
    `ptbWords` - A list of String
  Returns:
    A presentable version of the given PTB-tokenized words

### labelList2Text

    public static String labelList2Text(List<? extends HasWord> ptbWords)

  Returns a presentable version of the given PTB-tokenized words. Pass in a List of Words or a Document and this method will join the words with spaces and call `ptb2Text(String)` on the output. This method will take the word() values to prevent additional text from creeping in (e.g., POS tags).

  Parameters:
    `ptbWords` - A list of HasWord objects
  Returns:
    A presentable version of the given PTB-tokenized words

### factory

    public static TokenizerFactory<Word> factory()

### factory

    public static <T extends HasWord> TokenizerFactory<T> factory(boolean tokenizeNLs,
                                                                  LexedTokenFactory<T> factory)

### factory

    public static TokenizerFactory<CoreLabel> factory(boolean tokenizeNLs,
                                                      boolean invertible)

### factory

    public static <T extends HasWord> TokenizerFactory<T> factory(LexedTokenFactory<T> factory,
                                                                  String options)

### main

    public static void main(String[] args)
                     throws IOException

  Reads files named as arguments and print their tokens, by default as one per line. This is useful either for testing or to run standalone to turn a corpus into a one-token-per-line file of tokens. This main method assumes that the input file is in utf-8 encoding, unless it is specified.

  Usage: `java edu.stanford.nlp.process.PTBTokenizer [options] filename+`

  Options:
  - -options options Set various tokenization options (see the documentation in PTBLexer)
  - -preserveLines Produce space-separated tokens, except when the original had a line break, not one-token-per-line
  - -charset charset Specifies a character encoding
  - -parseInside regex Names an XML-style tag or a regular expression over such elements. The tokenizer will only tokenize inside element that match this name. (This is done by regex matching, not an XML parser, but works well for simply XML documents, or other SGML-style documents, such as Linguistic Data Consortium releases.)
  - -ioFileList file* The remaining command-line arguments are treated as filenames that themselves contain lists of pairs of input-output filenames (2 column, whitespace separated).
  - -dump Print the whole of each CoreLabel, not just the value (word)
  - -untok Heuristically untokenize tokenized text
  - -h Print usage info

  Parameters:
    `args` - Command line arguments
  Throws:
    `IOException` - If any file I/O problem

Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Stanford NLP Group