GeneralDataset (Stanford CoreNLP API)

java.lang.Object
- edu.stanford.nlp.classify.GeneralDataset<L,F>

Type Parameters:
L - The type of the labels in the Dataset
F - The type of the features in the Dataset

All Implemented Interfaces:

Serializable, Iterable<RVFDatum<L,F>>

Direct Known Subclasses:

Dataset, RVFDataset
```
public abstract class GeneralDataset<L,F>
extends Object
implements Serializable, Iterable<RVFDatum<L,F>>
```
The purpose of this interface is to unify Dataset and RVFDataset.

Author:

Kristina Toutanova (kristina@cs.stanford.edu), Anna Rafferty (various refactoring with subclasses), Sarah Spikes (sdspikes@cs.stanford.edu) (Templatization), Ramesh Nallapati (nmramesh@cs.stanford.edu) (added an abstract method getDatum, July 17th, 2008)

See Also:
Serialized Form

Field Summary

Fields
Modifier and Type Field and Description

protected int[][] data

Index<F> featureIndex

Index<L> labelIndex

protected int[] labels

protected int size

Fields
Modifier and Type	Field and Description
`protected int[][]`	`data`
`Index<F>`	`featureIndex`
`Index<L>`	`labelIndex`
`protected int[]`	`labels`
`protected int`	`size`

Constructor Summary

Constructors
Constructor and Description

GeneralDataset()

Constructors
Constructor and Description
`GeneralDataset()`

Method Summary

Methods
Modifier and Type	Method and Description
`abstract void`	`add(Datum<L,F> d)`
`void`	`addAll(Iterable<? extends Datum<L,F>> data)` Adds all Datums in the given collection of data to this dataset
`void`	`applyFeatureCountThreshold(int k)` Applies a feature count threshold to the Dataset.
`void`	`applyFeatureMaxCountThreshold(int k)` Applies a max feature count threshold to the Dataset.
`void`	`clear()` Resets the Dataset so that it is empty and ready to collect data.
`void`	`clear(int numDatums)` Resets the Dataset so that it is empty and ready to collect data.
`Index<F>`	`featureIndex()`
`int[][]`	`getDataArray()`
`abstract Datum<L,F>`	`getDatum(int index)`
`float[]`	`getFeatureCounts()` Get the total count (over all data instances) of each feature
`int[]`	`getLabelsArray()`
`abstract RVFDatum<L,F>`	`getRVFDatum(int index)`
`abstract double[][]`	`getValuesArray()`
`protected abstract void`	`initialize(int numDatums)` This method takes care of resetting values of the dataset such that it is empty with an initial capacity of numDatums.
`Iterator<RVFDatum<L,F>>`	`iterator()`
`Index<L>`	`labelIndex()`
`Iterator<L>`	`labelIterator()` Returns an iterator over the class labels of the Dataset
`String[]`	`makeSvmLabelMap()` Maps our labels to labels that are compatible with svm_light
`GeneralDataset<L,F>`	`mapDataset(GeneralDataset<L,F> dataset)`
`<L2> GeneralDataset<L2,F>`	`mapDataset(GeneralDataset<L,F> dataset, Index<L2> newLabelIndex, Map<L,L2> labelMapping, L2 defaultLabel)`
`static <L,L2,F> Datum<L2,F>`	`mapDatum(Datum<L,F> d, Map<L,L2> labelMapping, L2 defaultLabel)`
`int`	`numClasses()`
`ClassicCounter<L>`	`numDatumsPerLabel()`
`int`	`numFeatures()`
`int`	`numFeatureTokens()` returns the number of feature tokens in the Dataset.
`int`	`numFeatureTypes()` returns the number of distinct feature types in the Dataset.
`void`	`printSVMLightFormat()` Dumps the Dataset as a training/test file for SVMLight.
`void`	`printSVMLightFormat(PrintWriter pw)` Print SVM Light Format file.
`void`	`randomize(int randomSeed)` Randomizes the data array in place.
`GeneralDataset<L,F>`	`sampleDataset(int randomSeed, double sampleFrac, boolean sampleWithReplacement)`
`int`	`size()` Returns the number of examples (`Datum`s) in the Dataset.
`abstract Pair<GeneralDataset<L,F>,GeneralDataset<L,F>>`	`split(double p)`
`abstract Pair<GeneralDataset<L,F>,GeneralDataset<L,F>>`	`split(int start, int end)`
`abstract void`	`summaryStatistics()` Print some statistics summarizing the dataset
`protected void`	`trimData()`
`protected void`	`trimLabels()`
`protected double[][]`	`trimToSize(double[][] i)`
`protected int[]`	`trimToSize(int[] i)`
`protected int[][]`	`trimToSize(int[][] i)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - labelIndex
```
public Index<L> labelIndex
```
  - featureIndex
```
public Index<F> featureIndex
```
  - labels
```
protected int[] labels
```
  - data
```
protected int[][] data
```
  - size
```
protected int size
```
- Constructor Detail
  - GeneralDataset
```
public GeneralDataset()
```
- Method Detail
  - labelIndex
```
public Index<L> labelIndex()
```
  - featureIndex
```
public Index<F> featureIndex()
```
  - numFeatures
```
public int numFeatures()
```
  - numClasses
```
public int numClasses()
```
  - getLabelsArray
```
public int[] getLabelsArray()
```
  - getDataArray
```
public int[][] getDataArray()
```
  - getValuesArray
```
public abstract double[][] getValuesArray()
```
  - clear
```
public void clear()
```
    Resets the Dataset so that it is empty and ready to collect data.
  - clear
```
public void clear(int numDatums)
```
    Resets the Dataset so that it is empty and ready to collect data.
    
    Parameters:
    numDatums - initial capacity of dataset
  - initialize
```
protected abstract void initialize(int numDatums)
```
    This method takes care of resetting values of the dataset such that it is empty with an initial capacity of numDatums. Should be accessed only by appropriate methods within the class, such as clear(), which take care of other parts of the emptying of data.
    
    Parameters:
    numDatums - initial capacity of dataset
  - getRVFDatum
```
public abstract RVFDatum<L,F> getRVFDatum(int index)
```
  - getDatum
```
public abstract Datum<L,F> getDatum(int index)
```
  - add
```
public abstract void add(Datum<L,F> d)
```
  - getFeatureCounts
```
public float[] getFeatureCounts()
```
    Get the total count (over all data instances) of each feature
    
    Returns:
    an array containing the counts (indexed by index)
  - applyFeatureCountThreshold
```
public void applyFeatureCountThreshold(int k)
```
    Applies a feature count threshold to the Dataset. All features that occur fewer than k times are expunged.
  - applyFeatureMaxCountThreshold
```
public void applyFeatureMaxCountThreshold(int k)
```
    Applies a max feature count threshold to the Dataset. All features that occur greater than k times are expunged.
  - numFeatureTokens
```
public int numFeatureTokens()
```
    returns the number of feature tokens in the Dataset.
  - numFeatureTypes
```
public int numFeatureTypes()
```
    returns the number of distinct feature types in the Dataset.
  - addAll
```
public void addAll(Iterable<? extends Datum<L,F>> data)
```
    Adds all Datums in the given collection of data to this dataset
    
    Parameters:
    data - collection of datums you would like to add to the dataset
  - split
```
public abstract Pair<GeneralDataset<L,F>,GeneralDataset<L,F>> split(int start,
                                                  int end)
```
  - split
```
public abstract Pair<GeneralDataset<L,F>,GeneralDataset<L,F>> split(double p)
```
  - size
```
public int size()
```
    Returns the number of examples (Datums) in the Dataset.
  - trimData
```
protected void trimData()
```
  - trimLabels
```
protected void trimLabels()
```
  - trimToSize
```
protected int[] trimToSize(int[] i)
```
  - trimToSize
```
protected int[][] trimToSize(int[][] i)
```
  - trimToSize
```
protected double[][] trimToSize(double[][] i)
```
  - randomize
```
public void randomize(int randomSeed)
```
    Randomizes the data array in place. Note: this cannot change the values array or the datum weights, so redefine this for RVFDataset and WeightedDataset!
    
    Parameters:
    randomSeed -
  - sampleDataset
```
public GeneralDataset<L,F> sampleDataset(int randomSeed,
                                double sampleFrac,
                                boolean sampleWithReplacement)
```
  - summaryStatistics
```
public abstract void summaryStatistics()
```
    Print some statistics summarizing the dataset
  - labelIterator
```
public Iterator<L> labelIterator()
```
    Returns an iterator over the class labels of the Dataset
    
    Returns:
    An iterator over the class labels of the Dataset
  - mapDataset
```
public GeneralDataset<L,F> mapDataset(GeneralDataset<L,F> dataset)
```
    Parameters:
    dataset -
    
    Returns:
    a new GeneralDataset whose features and ids map exactly to those of this GeneralDataset. Useful when two Datasets are created independently and one wants to train a model on one dataset and test on the other. -Ramesh.
  - mapDatum
```
public static <L,L2,F> Datum<L2,F> mapDatum(Datum<L,F> d,
                            Map<L,L2> labelMapping,
                            L2 defaultLabel)
```
  - mapDataset
```
public <L2> GeneralDataset<L2,F> mapDataset(GeneralDataset<L,F> dataset,
                                   Index<L2> newLabelIndex,
                                   Map<L,L2> labelMapping,
                                   L2 defaultLabel)
```
    Parameters:
    dataset -
    
    Returns:
    a new GeneralDataset whose features and ids map exactly to those of this GeneralDataset. But labels are converted to be another set of labels
  - printSVMLightFormat
```
public void printSVMLightFormat()
```
    Dumps the Dataset as a training/test file for SVMLight.
    class [fno:val]+ The features must occur in consecutive order.
  - makeSvmLabelMap
```
public String[] makeSvmLabelMap()
```
    Maps our labels to labels that are compatible with svm_light
    
    Returns:
    array of strings
  - printSVMLightFormat
```
public void printSVMLightFormat(PrintWriter pw)
```
    Print SVM Light Format file. The following comments are no longer applicable because I am now printing out the exact labelID for each example. -Ramesh (nmramesh@cs.stanford.edu) 12/17/2009. If the Dataset has more than 2 classes, then it prints using the label index (+1) (for svm_struct). If it is 2 classes, then the labelIndex.get(0) is mapped to +1 and labelIndex.get(1) is mapped to -1 (for svm_light).
  - iterator
```
public Iterator<RVFDatum<L,F>> iterator()
```
    Specified by:
    
    iterator in interface Iterable<RVFDatum<L,F>>
  - numDatumsPerLabel
```
public ClassicCounter<L> numDatumsPerLabel()
```

Class GeneralDataset<L,F>

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

labelIndex

featureIndex

labels

data

size

Constructor Detail

GeneralDataset

Method Detail

labelIndex

featureIndex

numFeatures

numClasses

getLabelsArray

getDataArray

getValuesArray

clear

clear

initialize

getRVFDatum

getDatum

add

getFeatureCounts

applyFeatureCountThreshold

applyFeatureMaxCountThreshold

numFeatureTokens

numFeatureTypes

addAll

split

split

size

trimData

trimLabels

trimToSize

trimToSize

trimToSize

randomize

sampleDataset

summaryStatistics

labelIterator

mapDataset

mapDatum

mapDataset

printSVMLightFormat

makeSvmLabelMap

printSVMLightFormat

iterator

numDatumsPerLabel