edu.stanford.nlp.process

Class WordToSentenceProcessor<IN>

java.lang.Object

edu.stanford.nlp.process.WordToSentenceProcessor<IN>

Type Parameters:

IN - The type of the tokens in the sentences

All Implemented Interfaces:

ListProcessor<IN,java.util.List<IN>>

public class WordToSentenceProcessor<IN>
extends java.lang.Object
implements ListProcessor<IN,java.util.List<IN>>

Transforms a Document of Words into a Document of Sentences by grouping the Words. The word stream is assumed to already be adequately tokenized, and this class just divides the list into sentences, perhaps discarding some separator tokens based on the setting of the following three sets:

• sentenceBoundaryTokens are tokens that are left in a sentence, but are to be regarded as ending a sentence. A canonical example is a period. If two of these follow each other, the second will be a sentence consisting of only the sentenceBoundaryToken.
• sentenceBoundaryFollowers are tokens that are left in a sentence, and which can follow a sentenceBoundaryToken while still belonging to the previous sentence. They cannot begin a sentence (except at the beginning of a document). A canonical example is a close parenthesis ')'.
• sentenceBoundaryToDiscard are tokens which separate sentences and which should be thrown away. In web documents, a typical example would be a '<p>' tag. If two of these follow each other, they are coalesced: no empty Sentence is output. The end-of-file is not represented in this Set, but the code behaves as if it were a member.
• sentenceRegionBeginPattern A regular expression for marking the start of a sentence region. Not included in the sentence.
• sentenceRegionEndPattern A regular expression for marking the end of a sentence region. Not included in the sentence.

See DocumentPreprocessor for a class with a main method that will call this and cut a text file up into sentences.

Author:

Joseph Smarr (jsmarr@stanford.edu), Christopher Manning, Teg Grenager (grenager@stanford.edu), Sarah Spikes (sdspikes@cs.stanford.edu) (Templatization)

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.Set<java.lang.String>	DEFAULT_BOUNDARY_FOLLOWERS
static java.util.Set<java.lang.String>	DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

Constructor Summary

Constructors

Constructor and Description
WordToSentenceProcessor() Create a WordToSentenceProcessor using a sensible default list of tokens to split on for English/Latin writing systems.
WordToSentenceProcessor(java.lang.String boundaryTokenRegex) Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens and sentence boundary to discard tokens (based on English and Penn Treebank encoding).
WordToSentenceProcessor(java.lang.String boundaryTokenRegex, java.util.Set<java.lang.String> boundaryFollowers, java.util.Set<java.lang.String> boundaryToDiscard) Flexibly set the set of acceptable sentence boundary tokens, the set of tokens commonly following sentence boundaries, and also the set of tokens that are sentences boundaries that should be discarded.

Method Summary

Methods

Modifier and Type	Method and Description
void	addHtmlSentenceBoundaryToDiscard(java.util.Set<java.lang.String> set)
boolean	allowEmptySentences()
boolean	isOneSentence()
java.util.List<java.util.List<IN>>	process(java.util.List<? extends IN> words) Take a List (including a Sentence) of input, and return a List that has been processed in some way.
<L,F> Document<L,F,java.util.List<IN>>	processDocument(Document<L,F,IN> in)
void	setAllowEmptySentences(boolean allowEmptySentences)
void	setOneSentence(boolean oneSentence)
void	setSentenceBoundaryToDiscard(java.util.Set<java.lang.String> regexSet)
java.util.List<java.util.List<IN>>	wordsToSentences(java.util.List<? extends IN> words) Returns a List of Lists where each element is built from a run of Words in the input Document.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BOUNDARY_FOLLOWERS

public static final java.util.Set<java.lang.String> DEFAULT_BOUNDARY_FOLLOWERS

DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

public static final java.util.Set<java.lang.String> DEFAULT_SENTENCE_BOUNDARIES_TO_DISCARD

Constructor Detail

WordToSentenceProcessor

public WordToSentenceProcessor()

Create a WordToSentenceProcessor using a sensible default list of tokens to split on for English/Latin writing systems. The default set is: {".","?","!"} and any combination of ! or ?, as in !!!?!?!?!!!?!!?!!!.

WordToSentenceProcessor

public WordToSentenceProcessor(java.lang.String boundaryTokenRegex)

Flexibly set the set of acceptable sentence boundary tokens, but with a default set of allowed boundary following tokens and sentence boundary to discard tokens (based on English and Penn Treebank encoding). The allowed set of boundary followers is: {")","]","\"","\'", "''", "-RRB-", "-RSB-", "-RCB-"}. The default set of discarded separator tokens includes the newline tokens used by WhitespaceLexer and PTBLexer.

Parameters:

boundaryTokenRegex - The set of boundary tokens

WordToSentenceProcessor

public WordToSentenceProcessor(java.lang.String boundaryTokenRegex,
                       java.util.Set<java.lang.String> boundaryFollowers,
                       java.util.Set<java.lang.String> boundaryToDiscard)

Flexibly set the set of acceptable sentence boundary tokens, the set of tokens commonly following sentence boundaries, and also the set of tokens that are sentences boundaries that should be discarded.

Method Detail

setSentenceBoundaryToDiscard

public void setSentenceBoundaryToDiscard(java.util.Set<java.lang.String> regexSet)

isOneSentence

public boolean isOneSentence()

setOneSentence

public void setOneSentence(boolean oneSentence)

allowEmptySentences

public boolean allowEmptySentences()

setAllowEmptySentences

public void setAllowEmptySentences(boolean allowEmptySentences)

addHtmlSentenceBoundaryToDiscard

public void addHtmlSentenceBoundaryToDiscard(java.util.Set<java.lang.String> set)

process

public java.util.List<java.util.List<IN>> process(java.util.List<? extends IN> words)

Description copied from interface: ListProcessor

Take a List (including a Sentence) of input, and return a List that has been processed in some way.

Specified by:

process in interface ListProcessor<IN,java.util.List<IN>>

wordsToSentences

public java.util.List<java.util.List<IN>> wordsToSentences(java.util.List<? extends IN> words)

Returns a List of Lists where each element is built from a run of Words in the input Document. Specifically, reads through each word in the input document and breaks off a sentence after finding a valid sentence boundary token or end of file. Note that for this to work, the words in the input document must have been tokenized with a tokenizer that makes sentence boundary tokens their own tokens (e.g., PTBTokenizer).

Parameters:

words - A list of already tokenized words (must implement HasWord or be a String)

Returns:

A list of Sentence

See Also:

WordToSentenceProcessor(String, Set, Set, Pattern, Pattern)

processDocument

public <L,F> Document<L,F,java.util.List<IN>> processDocument(Document<L,F,IN> in)