edu.stanford.nlp.ie.machinereading.common
Class SimpleTokenize

java.lang.Object
  edu.stanford.nlp.ie.machinereading.common.SimpleTokenize

public class SimpleTokenize
extends java.lang.Object

Simple string tokenization

Constructor Summary

SimpleTokenize()

Method Summary

static void	main(java.lang.String[] argv) Implements a simple test
static java.lang.String	quotify(java.lang.String str) Constructs a valid quote-surrounded token All inside quotes are preceded by \
static java.util.ArrayList<java.lang.String>	tokenize(java.lang.String line) Basic string tokenization, skipping over white spaces
static java.util.ArrayList<java.lang.String>	tokenize(java.lang.String line, java.lang.String separators) Basic string tokenization, skipping over white spaces
static java.util.ArrayList<java.lang.String>	tokenizeWithQuotes(java.lang.String line) String tokenization, considering everything within quotes as 1 token Regular quotes inside tokens MUST be preceded by \

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleTokenize

public SimpleTokenize()

Method Detail

tokenize

public static java.util.ArrayList<java.lang.String> tokenize(java.lang.String line)

Basic string tokenization, skipping over white spaces

tokenize

public static java.util.ArrayList<java.lang.String> tokenize(java.lang.String line,
                                                             java.lang.String separators)

Basic string tokenization, skipping over white spaces

tokenizeWithQuotes

public static java.util.ArrayList<java.lang.String> tokenizeWithQuotes(java.lang.String line)

String tokenization, considering everything within quotes as 1 token Regular quotes inside tokens MUST be preceded by \

quotify

public static java.lang.String quotify(java.lang.String str)

Constructs a valid quote-surrounded token All inside quotes are preceded by \

main

public static void main(java.lang.String[] argv)

Implements a simple test