## edu.stanford.nlp.tagger.maxent
Class MaxentTagger

    java.lang.Object
      edu.stanford.nlp.tagger.maxent.MaxentTagger

All Implemented Interfaces:
  SentenceProcessor, ListProcessor<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>, Function<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>

    public class MaxentTagger

extends Object
implements Function<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>, SentenceProcessor, ListProcessor<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>

The main class for users to run, train, and test the part of speech tagger. You can tag things through the Java API or from the command line. The two English taggers included in this distribution are:

- A bi-directional dependency network tagger in models/bidirectional-distsim-wsj-0-18.tagger. Its accuracy was 97.32% on Penn Treebank WSJ secs. 22-24.
- A model using only left sequence information and similar but less unknown words and lexical features as the previous model in models/left3words-wsj-0-18.tagger. This tagger runs a lot faster. Its accuracy was 96.92% on Penn Treebank WSJ secs. 22-24.

### Using the Java API

A MaxentTagger can be made with a constructor taking as argument the location of parameter files for a trained tagger:
  `MaxentTagger tagger = new MaxentTagger("models/left3words-wsj-0-18.tagger");`

A default path is provided for the location of the tagger on the Stanford NLP machines:
  `MaxentTagger tagger = new MaxentTagger(DEFAULT_NLP_GROUP_MODEL_PATH);`

If you set the NLP_TAGGER_HOME environment variable, DEFAULT_NLP_GROUP_MODEL_PATH will instead point to the directory given in NLP_TAGGER_HOME.

To tag a Sentence and get a TaggedSentence:
  `Sentence taggedSentence = tagger.tagSentence(Sentence sentence)`
  `Sentence taggedSentence = tagger.apply(Sentence sentence)`

To tag a list of sentences and get back a list of tagged sentences:
  `List taggedList = tagger.process(List sentences)`

To tag a String of text and to get back a String with tagged words:
  `String taggedString = tagger.tagString("Here's a tagged string.")`

To tag a string of correctly tokenized, whitespace-separated words and get a string of tagged words back:
  `String taggedString = tagger.tagTokenizedString("Here 's a tagged string .")`

The `tagString` method uses the default tokenizer (PTBTokenizer). If you wish to control tokenization, you may wish to call `tokenizeText(Reader, TokenizerFactory)` and then to call `process()` on the result.

### Using the command line

Tagging, testing, and training can all also be done via the command line.

### Training from the command line

To train a model from the command line, first generate a property file:

    java edu.stanford.nlp.tagger.maxent.MaxentTagger -genprops

This gets you a default properties file with descriptions of each parameter you can set in your trained model. You can modify the properties file , or use the default options. To train, run:

    java -mx1g edu.stanford.nlp.tagger.maxent.MaxentTagger -props myPropertiesFile.props

with the appropriate properties file specified; any argument you give in the properties file can also be specified on the command line. You must have specified a model using -model, either in the properties file or on the command line, as well as a file containing tagged words using -trainFile. Useful flags for controlling the amount of output are -verbose, which prints extra debugging information, and -verboseResults, which prints full information about intermediate results. -verbose defaults to false and -verboseResults defaults to true.

### Tagging and Testing from the command line

Usage: For tagging (plain text):

    java edu.stanford.nlp.tagger.maxent.MaxentTagger -model <modelFile> -textFile <textfile>

For testing (evaluating against tagged text):

    java edu.stanford.nlp.tagger.maxent.MaxentTagger -model <modelFile> -testFile <testfile>

You can use the same properties file as for training if you pass it in with the "-props" argument. The most important arguments for tagging (besides "model" and "file") are "tokenize" and "tokenizerFactory". See below for more details. Note that the tagger assumes input has not yet been tokenized and by default tokenizes it using a default English tokenizer. If your input has already been tokenized, use the flag "-tokenized".

Parameters can be defined using a Properties file (specified on the command-line with `-prop` propFile), or directly on the command line (by preceding their name with a minus sign ("-") to turn them into a flag. The following properties are recognized:

Property Name | Type | Default Value | Relevant Phase(s) | Description
model | String | N/A | All | Path and filename where you would like to save the model (training) or where the model should be loaded from (testing, tagging).
trainFile | String | N/A | Train | Path to the file holding the training data; specifying this option puts the tagger in training mode. Only one of 'trainFile','testFile','texFile', and 'convertToSingleFile' may be specified.
testFile | String | N/A | Test | Path to the file holding the test data; specifying this option puts the tagger in testing mode. Only one of 'trainFile','testFile','texFile', and 'convertToSingleFile' may be specified.
textFile | String | N/A | Tag | Path to the file holding the text to tag; specifying this option puts the tagger in tagging mode. Only one of 'trainFile','testFile','textFile', and 'convertToSingleFile' may be specified.

convertToSingleFile

String

N/A

N/A

Provided only for backwards compatibility, this option allows you to convert a tagger trained using a previous version of the tagger to the new single-file format. The value of this flag should be the path for the new model file, 'model' should be the path prefix to the old tagger (up to but not including the ".holder"), and you should supply the properties configuration for the old tagger with -props (before these two arguments).

genprops | boolean | N/A | N/A | Use this option to output a default properties file, containing information about each of the possible configuration options.
delimiter | char | / | All | Delimiter character that separates word and part of speech tags. For training and testing, this is the delimiter used in the train/test files. For tagging, this is the character that will be inserted between words and tags in the output.
encoding | String | UTF-8 | All | Encoding of the read files (training, testing) and the output text files.
tokenize | boolean | true | Tag,Test | Whether or not the file has been tokenized. If this is true, the tagger assumes that white space separates all and only those things that should be tagged as separate tokens, and that the input is strictly one sentence per line.
tokenizerFactory | String | edu.stanford.nlp.process.PTBTokenizer | Tag,Test | Fully qualified class name of the tokenizer to use. edu.stanford.nlp.process.PTBTokenizer does basic English tokenization.
tokenizerOptions | String |  | Tag,Test | Known options for the particular tokenizer used. A comma-separated list. For PTBTokenizer, options of interest include `americanize=false` and `asciiQuotes` (for German). Note that any choice of tokenizer options that conflicts with the tokenization used in the tagger training data will likely degrade tagger performance.

arch

String

generic

Train

Architecture of the model, as a comma-separated list of options, some with a parenthesized integer argument written k here: this determines what features are sed to build your model. Options are 'left3words', 'left5words', 'bidirectional', 'bidirectional5words', generic', 'sighan2005' (Chinese), 'german', 'words(k),' 'naacl2003unknowns', 'naacl2003conjunctions', wordshapes(k), motleyUnknown, suffix(k), prefix(k), prefixsuffix(k), capitalizationsuffix(k), distsim(s), chinesedictionaryfeatures(s), lctagfeatures, unicodeshapes(k). The left3words architectures are faster, but slightly less accurate, than the bidirectional architectures. 'naacl2003unknowns' was our traditional set of unknown word features, but you can now specify features more flexibility via the various other supported keywords. The 'shapes' options map words to equivalence classes, which slightly increase accuracy.

lang | String | english | Train | Language from which the part of speech tags are drawn. This option determines which tags are considered closed-class (only fixed set of words can be tagged with a closed-class tag, such as prepositions). Defined languages are 'english' (Penn tagset), 'polish' (very rudimentary), 'chinese', 'arabic', 'german', and 'medline'.
openClassTags | String | N/A | Train | Space separated list of tags that should be considered open-class. All tags encountered that are not in this list are considered closed-class. E.g. format: "NN VB"
closedClassTags | String | N/A | Train | Space separated list of tags that should be considered closed-class. All tags encountered that are not in this list are considered open-class.
learnClosedClassTags | boolean | false | Train | If true, induce which tags are closed-class by counting as closed-class tags all those tags which have fewer unique word tokens than closedClassTagThreshold.
closedClassTagThreshold | int | int | Train | Number of unique word tokens that a tag may have and still be considered closed-class; relevant only if learnClosedClassTags is true.
sgml | boolean | false | Tag, Test | Very basic tagging of the contents of all sgml fields; for more complex mark-up, consider using the xmlInput option.
xmlInput | String |  | Tag, Test | Give a space separated list of tags in an XML file whose content you would like tagged. Any internal tags that appear in the content of fields you would like tagged will be discarded; the rest of the XML will be preserved and the original text of specified fields will be replaced with the tagged text.

xmlOutput

String

""

Tag

If a path is given, the tagged data be written out to the given file in xml. If non-empty, each word will be written out within a word tag, with the part of speech as an attribute. If original input was XML, this will just appear in the field where the text originally came from. Otherwise, word tags will be surrounded by sentence tags as well. E.g., <sentence id="0"><word id="0" pos="NN">computer</word></sentence>

tagInside | String | "" | Tag | Tags inside elements that match the regular expression given in the String.
search | String | cg | Train | Specify the search method to be used in the optimization method for training. Options are 'cg' (conjugate gradient) or 'iis' (improved iterative scaling).
sigmaSquared | double | 0.5 | Train | Sigma-squared smoothing/regularization parameter to be used for conjugate gradient search. Default usually works reasonably well.
iterations | int | 100 | Train | Number of iterations to be used for improved iterative scaling.
rareWordThresh | int | 5 | Train | Words that appear fewer than this number of times during training are considered rare words and use extra rare word features.
minFeatureThreshold | int | 5 | Train | Features whose history appears fewer than this number of times are discarded.
curWordMinFeatureThreshold | int | 2 | Train | Words that occur more than this number of times will generate features with all of the tags they've been seen with.
rareWordMinFeatureThresh | int | 10 | Train | Features of rare words whose histories occur fewer than this number of times are discarded.
veryCommonWordThresh | int | 250 | Train | Words that occur more than this number of times form an equivalence class by themselves. Ignored unless you are using ambiguity classes.
debug | boolean | boolean | All | Whether to write debugging information (words, top words, unknown words). Useful for error analysis.
debugPrefix | String | N/A | All | File (path) prefix for where to write out the debugging information (relevant only if debug=true).

Author:
  Kristina Toutanova, Miler Lee, Joseph Smarr, Anna Rafferty, Michel Galley, Christopher Manning, John Bauer

Field Summary
`static String` `DEFAULT_DISTRIBUTION_PATH`
`static String` `DEFAULT_NLP_GROUP_MODEL_PATH`
`static String` `TAGGER_HOME` The directory from which to get taggers when using DEFAULT_NLP_GROUP_MODEL_PATH.

Constructor Summary
`MaxentTagger()`
`MaxentTagger(String modelFile)` Constructor for a tagger using a model stored in a particular file.
`MaxentTagger(String modelFile, TaggerConfig config)` Constructor for a tagger using a model stored in a particular file, with options taken from the supplied TaggerConfig.
`MaxentTagger(String modelFile, TaggerConfig config, boolean printLoading)` Initializer that loads the tagger.

Method Summary
`ArrayList<TaggedWord>` `apply(ArrayList<? extends HasWord> in)` Expects a sentence and returns a tagged sentence.
`protected TokenizerFactory<? extends HasWord>` `chooseTokenizerFactory()` Figures out what tokenizer factory might be described by the config.
`protected static TokenizerFactory<? extends HasWord>` `chooseTokenizerFactory(boolean tokenize, String tokenizerFactory, String tokenizerOptions)`
`protected void` `dumpModel()`
`TTags` `getTags()`
`static void` `main(String[] args)` Command-line tagger interface.
`List<ArrayList<TaggedWord>>` `process(List<? extends ArrayList<? extends HasWord>> sentences)` Tags the Words in each Sentence in the given List with their grammatical part-of-speech.
`ArrayList<TaggedWord>` `processSentence(ArrayList sentence)` Returns a new Sentence that is a copy of the given sentence with all the words tagged with their part-of-speech.
`protected void` `readModelAndInit(TaggerConfig config, DataInputStream rf, boolean printLoading)` This reads the complete tagger from a single model file, and inits the tagger using a combination of the properties passed in and parameters from the file.
`protected void` `readModelAndInit(TaggerConfig config, String modelFileOrUrl, boolean printLoading)` This reads the complete tagger from a single model stored in a file, at a URL, or as a resource in a jar file, and inits the tagger using a combination of the properties passed in and parameters from the file.
`void` `runTagger(BufferedReader reader, BufferedWriter writer, String tagInside, boolean stdin)` This method runs the tagger on the provided reader & writer.
`protected void` `saveModel(String filename, TaggerConfig config)`
`void` `tagFromXML(InputStream input, Writer writer, String... xmlTags)` Uses an XML transformer to turn an input stream into a bunch of output.
`void` `tagFromXML(Reader input, Writer writer, String... xmlTags)`
`ArrayList<TaggedWord>` `tagSentence(List<? extends HasWord> sentence)` Returns a new Sentence that is a copy of the given sentence with all the words tagged with their part-of-speech.
`String` `tagString(String toTag)` Tags the input string and returns the tagged version.
`String` `tagTokenizedString(String toTag)` Tags the tokenized input string and returns the tagged version.
`static List<ArrayList<? extends HasWord>>` `tokenizeText(Reader r)` Reads data from r, tokenizes it with the default (Penn Treebank) tokenizer, and returns a List of Sentence objects, which can then be fed into tagSentence.
`protected static List<ArrayList<? extends HasWord>>` `tokenizeText(Reader r, TokenizerFactory<? extends HasWord> tokenizerFactory)` Reads data from r, tokenizes it with the given tokenizer, and returns a List of Lists of (extends) HasWord objects, which can then be fed into tagSentence.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### TAGGER_HOME

    public static final String TAGGER_HOME

  The directory from which to get taggers when using DEFAULT_NLP_GROUP_MODEL_PATH. Normally set to the location of the latest left3words tagger on the NLP machines, but can be changed by setting the environment variable TAGGER_HOME.

### DEFAULT_NLP_GROUP_MODEL_PATH

    public static final String DEFAULT_NLP_GROUP_MODEL_PATH

### DEFAULT_DISTRIBUTION_PATH

    public static final String DEFAULT_DISTRIBUTION_PATH

See Also:
  Constant Field Values

Constructor Detail

### MaxentTagger

    public MaxentTagger()

### MaxentTagger

    public MaxentTagger(String modelFile)
                 throws IOException,
                        ClassNotFoundException

  Constructor for a tagger using a model stored in a particular file. The `modelFile` is a filename for the model data. The tagger data is loaded when the constructor is called (this can be slow). This constructor first constructs a TaggerConfig object, which loads the tagger options from the modelFile.

  Parameters:
    `modelFile` - filename of the trained model
  Throws:
    `IOException` - if IO problem
    `ClassNotFoundException` - when there are errors loading a tagger

### MaxentTagger

    public MaxentTagger(String modelFile,
                        TaggerConfig config)
                 throws IOException,
                        ClassNotFoundException

  Constructor for a tagger using a model stored in a particular file, with options taken from the supplied TaggerConfig. The `modelFile` is a filename for the model data. The tagger data is loaded when the constructor is called (this can be slow). This version assumes that the tagger options in the modelFile have already been loaded into the TaggerConfig (if that is desired).

  Parameters:
    `modelFile` - filename of the trained model
    `config` - The configuration for the tagger
  Throws:
    `IOException` - if IO problem
    `ClassNotFoundException` - when there are errors loading a tagger

### MaxentTagger

    public MaxentTagger(String modelFile,
                        TaggerConfig config,
                        boolean printLoading)
                 throws IOException,
                        ClassNotFoundException

  Initializer that loads the tagger.

  Parameters:
    `modelFile` - Where to initialize the tagger from. Most commonly, this is the filename of the trained model, for example, `/u/nlp/data/pos-tagger/wsj3t0-18-left3words/left3words-wsj-0-18.tagger` . However, if it starts with "https?://" it will be interpreted as a URL, and if it starts with "jar:" it will be taken as a resources in the /models/ path of the current jar file.
    `config` - TaggerConfig based on command-line arguments
    `printLoading` - Whether to print a message saying what model file is being loaded and how long it took when finished.
  Throws:
    `IOException` - if IO problem
    `ClassNotFoundException` - when there are errors loading a tagger

Method Detail

### chooseTokenizerFactory

    protected TokenizerFactory<? extends HasWord> chooseTokenizerFactory()
                                                                  throws ClassNotFoundException,
                                                                         NoSuchMethodException,
                                                                         IllegalAccessException,
                                                                         InvocationTargetException

  Figures out what tokenizer factory might be described by the config. If it's described by name in the config, uses reflection to get the factory (which may cause an exception, of course...)

  Throws:
    `ClassNotFoundException`
    `NoSuchMethodException`
    `IllegalAccessException`
    `InvocationTargetException`

### chooseTokenizerFactory

    protected static TokenizerFactory<? extends HasWord> chooseTokenizerFactory(boolean tokenize,
                                                                                String tokenizerFactory,
                                                                                String tokenizerOptions)
                                                                         throws ClassNotFoundException,
                                                                                NoSuchMethodException,
                                                                                IllegalAccessException,
                                                                                InvocationTargetException

  Throws:
    `ClassNotFoundException`
    `NoSuchMethodException`
    `IllegalAccessException`
    `InvocationTargetException`

### saveModel

    protected void saveModel(String filename,
                             TaggerConfig config)

### readModelAndInit

    protected void readModelAndInit(TaggerConfig config,
                                    String modelFileOrUrl,
                                    boolean printLoading)
                             throws IOException,
                                    ClassNotFoundException

  This reads the complete tagger from a single model stored in a file, at a URL, or as a resource in a jar file, and inits the tagger using a combination of the properties passed in and parameters from the file.

  Note for the future: This assumes that the TaggerConfig in the file has already been read and used. This work is done inside the constructor of TaggerConfig. It might be better to refactor things so that is all done inside this method, but for the moment it seemed better to leave working code alone [cdm 2008].

  Parameters:
    `config` - The tagger config
    `modelFileOrUrl` - The name of the model file. This routine opens and closes it.
    `printLoading` - Whether to print a message saying what model file is being loaded and how long it took when finished.
  Throws:
    `IOException` - If I/O errors, etc.
    `ClassNotFoundException` - especially for incompatible tagger formats

### readModelAndInit

    protected void readModelAndInit(TaggerConfig config,
                                    DataInputStream rf,
                                    boolean printLoading)
                             throws IOException,
                                    ClassNotFoundException

  This reads the complete tagger from a single model file, and inits the tagger using a combination of the properties passed in and parameters from the file.

  Note for the future: This assumes that the TaggerConfig in the file has already been read and used. It might be better to refactor things so that is all done inside this method, but for the moment it seemed better to leave working code alone [cdm 2008].

  Parameters:
    `config` - The tagger config
    `rf` - DataInputStream to read from. It's the caller's job to open and close this stream.
    `printLoading` - Whether to print a message saying what model file is being loaded and how long it took when finished.
  Throws:
    `IOException` - If I/O errors
    `ClassNotFoundException` - If serialization errors

### dumpModel

    protected void dumpModel()

### getTags

    public TTags getTags()

### tagTokenizedString

    public String tagTokenizedString(String toTag)

  Tags the tokenized input string and returns the tagged version. This method requires the input to already be tokenized. The tagger wants input that is whitespace separated tokens, tokenized according to the conventions of the training data. (For instance, for the Penn Treebank, punctuation marks and possessive "'s" should be separated from words.)

  Parameters:
    `toTag` - The untagged input String
  Returns:
    The same string with tags inserted in the form word/tag

### tagString

    public String tagString(String toTag)

  Tags the input string and returns the tagged version. This method tokenizes the input into words in perhaps multiple sentences and then tags those sentences. The default (PTB English) tokenizer is used.

  Parameters:
    `toTag` - The untagged input String
  Returns:
    A String of sentences with tags inserted in the form word/tag

### apply

    public ArrayList<TaggedWord> apply(ArrayList<? extends HasWord> in)

  Expects a sentence and returns a tagged sentence. The input Sentence items

  Specified by:
    `apply` in interface `Function<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>`
  Parameters:
    `in` - This needs to be a Sentence
  Returns:
    A Sentence of TaggedWord

### process

    public List<ArrayList<TaggedWord>> process(List<? extends ArrayList<? extends HasWord>> sentences)

  Tags the Words in each Sentence in the given List with their grammatical part-of-speech. The returned List contains Sentences consisting of TaggedWords.

  NOTE: The input document must contain sentences as its elements, not words. To turn a Document of words into a Document of sentences, run it through `WordToSentenceProcessor`.

  Specified by:
    `process` in interface `ListProcessor<ArrayList<? extends HasWord>,ArrayList<TaggedWord>>`
  Parameters:
    `sentences` - A List of Sentence
  Returns:
    A List of Sentence of TaggedWord (final generification cannot be listed due to lack of complete generification of super classes)

### processSentence

    public ArrayList<TaggedWord> processSentence(ArrayList sentence)

  Returns a new Sentence that is a copy of the given sentence with all the words tagged with their part-of-speech. Convenience method when you only want to tag a single Sentence instead of a Document of sentences.

  Specified by:
    `processSentence` in interface `SentenceProcessor`
  Parameters:
    `sentence` - A sentence. Classes implementing this interface can assume that the sentence passed in is not null.

### tagSentence

    public ArrayList<TaggedWord> tagSentence(List<? extends HasWord> sentence)

  Returns a new Sentence that is a copy of the given sentence with all the words tagged with their part-of-speech. Convenience method when you only want to tag a single Sentence instead of a Document of sentences.

  Parameters:
    `sentence` - sentence to tag
  Returns:
    tagged sentence

### tokenizeText

    public static List<ArrayList<? extends HasWord>> tokenizeText(Reader r)

  Reads data from r, tokenizes it with the default (Penn Treebank) tokenizer, and returns a List of Sentence objects, which can then be fed into tagSentence.

  Parameters:
    `r` - Reader where untokenized text is read
  Returns:
    List of tokenized sentences

### tokenizeText

    protected static List<ArrayList<? extends HasWord>> tokenizeText(Reader r,
                                                                     TokenizerFactory<? extends HasWord> tokenizerFactory)

  Reads data from r, tokenizes it with the given tokenizer, and returns a List of Lists of (extends) HasWord objects, which can then be fed into tagSentence.

  Parameters:
    `r` - Reader where untokenized text is read
    `tokenizerFactory` - Tokenizer. This can be `null` in which case the default English tokenizer (PTBTokenizerFactory) is used.
  Returns:
    List of tokenized sentences

### tagFromXML

    public void tagFromXML(InputStream input,
                           Writer writer,
                           String... xmlTags)

  Uses an XML transformer to turn an input stream into a bunch of output. Tags all of the text between xmlTags. The difference between using this and using runTagger in XML mode is that this preserves the XML structure outside of the list of elements to tag, whereas the runTagger method throws away all of the surrounding structure and returns tagged plain text.

### tagFromXML

    public void tagFromXML(Reader input,
                           Writer writer,
                           String... xmlTags)

### runTagger

    public void runTagger(BufferedReader reader,
                          BufferedWriter writer,
                          String tagInside,
                          boolean stdin)
                   throws IOException,
                          ClassNotFoundException,
                          NoSuchMethodException,
                          IllegalAccessException,
                          InvocationTargetException

  This method runs the tagger on the provided reader & writer. It takes into from the given `reader`, applies the tagger to it one sentence at a time (determined using documentPreprocessor), and writes the output to the given `writer`. The document is broken into sentences using the sentence processor determined in the tagger's TaggerConfig. `tagInside` makes the tagger run in XML mode... if set to non-empty, instead of processing the document as one large text blob, it considers each region in between the given tag to be a separate text blob. `stdin` makes the tagger take lines one at a time from the reader and print out a bit more nice formatting to the writer. It doesn't actually require the reader to be stdin, though. This is mutually exclusive with tagInside; if tagInside is not empty, stdin will be ignored.

  Throws:
    `IOException`
    `ClassNotFoundException`
    `NoSuchMethodException`
    `IllegalAccessException`
    `InvocationTargetException`

### main

    public static void main(String[] args)
                     throws Exception

  Command-line tagger interface. Can be used to train or test taggers, or to tag text, taking input from stdin or a file. See class documentation for usage.

  Parameters:
    `args` - Command-line arguments
  Throws:
    `IOException` - If any file problems
    `Exception`

Stanford NLP Group