edu.stanford.nlp.ie.machinereading.domains.ace.reader
Class RobustTokenizer<T extends Word>

java.lang.Object
  edu.stanford.nlp.process.AbstractTokenizer<Word>
      edu.stanford.nlp.ie.machinereading.domains.ace.reader.RobustTokenizer<T>

All Implemented Interfaces:

Tokenizer<Word>, java.util.Iterator<Word>

public class RobustTokenizer<T extends Word>
extends AbstractTokenizer<Word>

Nested Class Summary

static class	RobustTokenizer.AbbreviationMap
static class	RobustTokenizer.WordToken

Field Summary

static java.lang.String	ACRONYM
static java.lang.String	AMPERSAND
static java.lang.String	ANY
static java.lang.String	APOSTROPHE
static java.lang.String	AT
static java.lang.String	BLOCK
static java.lang.String	COMMA
static java.lang.String	DATE
static java.lang.String	DECNUM
static java.lang.String	DIGIT
static java.lang.String	DIGITSEQ
static java.lang.String	DOMAIN_EMAIL
static java.lang.String	DOT
static java.lang.String	DOTCOMMA
static java.lang.String	DOTDOT
static java.lang.String	DOUBLE_QUOTES
static java.lang.String	EMAIL
static java.lang.String	FULLNUM
static java.lang.String	GREATER
static java.lang.String	HTMLCODE
static java.lang.String	HTTP
static java.lang.String	LCB
static java.lang.String	LETTER
static java.lang.String	LETTERS
static java.lang.String	LIST_BULLET
static java.lang.String	LOOSE_ACRONYM
static java.lang.String	LOWER
static java.lang.String	LRB
static int	MAX_MULTI_WORD_SIZE
static java.lang.String	MINUS
static java.lang.String	NUM
static java.lang.String	PAREN
static java.lang.String	PHONE_PART
static java.lang.String	PLUS
static java.lang.String	PUNC
static java.lang.String	QUOTES
static java.lang.String	RCB
static java.lang.String	RECOGNISED_PATTERN
static java.lang.String	RRB
static java.lang.String	SGML
static java.lang.String	SIGN
static java.lang.String	SLASH
static java.lang.String	SMALL_URL
static java.lang.String	TIME
static java.lang.String	UNDERSCORE
static java.lang.String	UNDERSCORESEQ
static java.lang.String	UPPER
static java.lang.String	URL
static java.lang.String	WHITE_SPACE
static java.lang.String	WORD

Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer

nextToken

Constructor Summary

RobustTokenizer(boolean caseInsensitive, java.lang.String buffer)

RobustTokenizer(java.lang.String buffer)

Method Summary

static java.lang.String	block(java.lang.String s) parens
int	countNewLines(java.lang.String s, int start, int end)
protected Word	getNext() Internally fetches the next token.
static boolean	isAcronym(java.lang.String s)
static boolean	isDigitSeq(java.lang.String s)
static boolean	isEmail(java.lang.String s)
static boolean	isSgml(java.lang.String s)
static boolean	isSlashDate(java.lang.String s)
static boolean	isUrl(java.lang.String s)
static void	main(java.lang.String[] argv)
static java.lang.String	oneOrMore(java.lang.String s) one or more
static java.lang.String	or(java.lang.String s1, java.lang.String s2) any of the two
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3) any of the three
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4) any of the four
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5) any of the five
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6) any of the six
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7) any of the seven
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7, java.lang.String s8) any of the eight
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7, java.lang.String s8, java.lang.String s9) any of the nine
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7, java.lang.String s8, java.lang.String s9, java.lang.String s10)
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7, java.lang.String s8, java.lang.String s9, java.lang.String s10, java.lang.String s11)
static java.lang.String	or(java.lang.String s1, java.lang.String s2, java.lang.String s3, java.lang.String s4, java.lang.String s5, java.lang.String s6, java.lang.String s7, java.lang.String s8, java.lang.String s9, java.lang.String s10, java.lang.String s11, java.lang.String s12)
protected java.util.List<RobustTokenizer.WordToken>	postprocess(java.util.List<RobustTokenizer.WordToken> tokens) Redefine this method to implement additional domain-specific tokenization rules
static java.lang.String	range(java.lang.String s) any in the set
static java.lang.String	rangeNot(java.lang.String s) not
java.lang.String	tokenizeText() Tokenizes and adds blank spaces were needed between each token
Word[]	tokenizeToWords() Smart tokenization storing the output in an array of CoreLabel Sets the following fields: - TextAnnotation - the text of the token - TokenBeginAnnotation - the byte offset of the token (start) - TokenEndAnnotation - the byte offset of the token (end)
java.util.List<RobustTokenizer.WordToken>	tokenizeToWordTokens() Tokenizes a natural language string
static java.lang.String	zeroOrMore(java.lang.String s) zero or more
static java.lang.String	zeroOrOne(java.lang.String s) zero or one

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer

hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_MULTI_WORD_SIZE

public static final int MAX_MULTI_WORD_SIZE

See Also:

Constant Field Values

DOT

public static final java.lang.String DOT

DOTDOT

public static final java.lang.String DOTDOT

APOSTROPHE

public static final java.lang.String APOSTROPHE

SLASH

public static final java.lang.String SLASH

UNDERSCORE

public static final java.lang.String UNDERSCORE

MINUS

public static final java.lang.String MINUS

PLUS

public static final java.lang.String PLUS

COMMA

public static final java.lang.String COMMA

DOTCOMMA

public static final java.lang.String DOTCOMMA

QUOTES

public static final java.lang.String QUOTES

DOUBLE_QUOTES

public static final java.lang.String DOUBLE_QUOTES

LRB

public static final java.lang.String LRB

RRB

public static final java.lang.String RRB

LCB

public static final java.lang.String LCB

RCB

public static final java.lang.String RCB

GREATER

public static final java.lang.String GREATER

LOWER

public static final java.lang.String LOWER

AMPERSAND

public static final java.lang.String AMPERSAND

AT

public static final java.lang.String AT

HTTP

public static final java.lang.String HTTP

WHITE_SPACE

public static final java.lang.String WHITE_SPACE

DIGIT

public static final java.lang.String DIGIT

LETTER

public static final java.lang.String LETTER

UPPER

public static final java.lang.String UPPER

SIGN

public static final java.lang.String SIGN

FULLNUM

public static final java.lang.String FULLNUM

DECNUM

public static final java.lang.String DECNUM

NUM

public static final java.lang.String NUM

DATE

public static final java.lang.String DATE

TIME

public static final java.lang.String TIME

PUNC

public static final java.lang.String PUNC

LETTERS

public static final java.lang.String LETTERS

BLOCK

public static final java.lang.String BLOCK

WORD

public static final java.lang.String WORD

ACRONYM

public static final java.lang.String ACRONYM

LOOSE_ACRONYM

public static final java.lang.String LOOSE_ACRONYM

PAREN

public static final java.lang.String PAREN

SGML

public static final java.lang.String SGML

See Also:

Constant Field Values

HTMLCODE

public static final java.lang.String HTMLCODE

ANY

public static final java.lang.String ANY

EMAIL

public static final java.lang.String EMAIL

DOMAIN_EMAIL

public static final java.lang.String DOMAIN_EMAIL

URL

public static final java.lang.String URL

SMALL_URL

public static final java.lang.String SMALL_URL

UNDERSCORESEQ

public static final java.lang.String UNDERSCORESEQ

LIST_BULLET

public static final java.lang.String LIST_BULLET

PHONE_PART

public static final java.lang.String PHONE_PART

DIGITSEQ

public static final java.lang.String DIGITSEQ

RECOGNISED_PATTERN

public static final java.lang.String RECOGNISED_PATTERN

Constructor Detail

RobustTokenizer

public RobustTokenizer(java.lang.String buffer)

RobustTokenizer

public RobustTokenizer(boolean caseInsensitive,
                       java.lang.String buffer)

Method Detail

range

public static java.lang.String range(java.lang.String s)

any in the set

zeroOrOne

public static java.lang.String zeroOrOne(java.lang.String s)

zero or one

zeroOrMore

public static java.lang.String zeroOrMore(java.lang.String s)

zero or more

oneOrMore

public static java.lang.String oneOrMore(java.lang.String s)

one or more

block

public static java.lang.String block(java.lang.String s)

parens

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2)

any of the two

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3)

any of the three

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4)

any of the four

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5)

any of the five

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6)

any of the six

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7)

any of the seven

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7,
                                  java.lang.String s8)

any of the eight

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7,
                                  java.lang.String s8,
                                  java.lang.String s9)

any of the nine

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7,
                                  java.lang.String s8,
                                  java.lang.String s9,
                                  java.lang.String s10)

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7,
                                  java.lang.String s8,
                                  java.lang.String s9,
                                  java.lang.String s10,
                                  java.lang.String s11)

or

public static java.lang.String or(java.lang.String s1,
                                  java.lang.String s2,
                                  java.lang.String s3,
                                  java.lang.String s4,
                                  java.lang.String s5,
                                  java.lang.String s6,
                                  java.lang.String s7,
                                  java.lang.String s8,
                                  java.lang.String s9,
                                  java.lang.String s10,
                                  java.lang.String s11,
                                  java.lang.String s12)

rangeNot

public static java.lang.String rangeNot(java.lang.String s)

not

isUrl

public static boolean isUrl(java.lang.String s)

isEmail

public static boolean isEmail(java.lang.String s)

isSgml

public static boolean isSgml(java.lang.String s)

isSlashDate

public static boolean isSlashDate(java.lang.String s)

isAcronym

public static boolean isAcronym(java.lang.String s)

isDigitSeq

public static boolean isDigitSeq(java.lang.String s)

countNewLines

public int countNewLines(java.lang.String s,
                         int start,
                         int end)

tokenizeToWords

public Word[] tokenizeToWords()

Smart tokenization storing the output in an array of CoreLabel Sets the following fields: - TextAnnotation - the text of the token - TokenBeginAnnotation - the byte offset of the token (start) - TokenEndAnnotation - the byte offset of the token (end)

tokenizeToWordTokens

public java.util.List<RobustTokenizer.WordToken> tokenizeToWordTokens()

Tokenizes a natural language string

Returns:

List of WordTokens

postprocess

protected java.util.List<RobustTokenizer.WordToken> postprocess(java.util.List<RobustTokenizer.WordToken> tokens)

Redefine this method to implement additional domain-specific tokenization rules

Parameters:

tokens -

tokenizeText

public java.lang.String tokenizeText()
                              throws java.io.IOException

Tokenizes and adds blank spaces were needed between each token

Throws:

java.io.IOException

getNext

protected Word getNext()

Description copied from class: AbstractTokenizer

Internally fetches the next token.

Specified by:

getNext in class AbstractTokenizer<Word>

Returns:

the next token in the token stream, or null if none exists.

main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception

Throws:

java.lang.Exception