com.rapidminer.operator.performance
Class AbstractPerformanceEvaluator

java.lang.Object
  com.rapidminer.operator.Operator
      com.rapidminer.operator.performance.AbstractPerformanceEvaluator

All Implemented Interfaces:

ConfigurationListener, PreviewListener, ParameterHandler, LoggingHandler

Direct Known Subclasses:

BinominalClassificationPerformanceEvaluator, ForecastingPerformanceEvaluator, PerformanceEvaluator, PolynominalClassificationPerformanceEvaluator, RegressionPerformanceEvaluator, SimplePerformanceEvaluator, UserBasedPerformanceEvaluator

public abstract class AbstractPerformanceEvaluator
extends Operator

This performance evaluator operator should be used for regression tasks, i.e. in cases where the label attribute has a numerical value type. The operator expects a test ExampleSet as input, whose elements have both true and predicted labels, and delivers as output a list of performance values according to a list of performance criteria that it calculates. If an input performance vector was already given, this is used for keeping the performance values.

All of the performance criteria can be switched on using boolean parameters. Their values can be queried by a ProcessLogOperator using the same names. The main criterion is used for comparisons and need to be specified only for processes where performance vectors are compared, e.g. feature selection or other meta optimization process setups. If no other main criterion was selected, the first criterion in the resulting performance vector will be assumed to be the main criterion.

The resulting performance vectors are usually compared with a standard performance comparator which only compares the fitness values of the main criterion. Other implementations than this simple comparator can be specified using the parameter comparator_class. This may for instance be useful if you want to compare performance vectors according to the weighted sum of the individual criteria. In order to implement your own comparator, simply subclass PerformanceComparator. Please note that for true multi-objective optimization usually another selection scheme is used instead of simply replacing the performance comparator.

Author:

Ingo Mierswa

Field Summary

static java.lang.String	PARAMETER_COMPARATOR_CLASS The parameter name for "Fully qualified classname of the PerformanceComparator implementation.
static java.lang.String	PARAMETER_MAIN_CRITERION The parameter name for "The criterion used for comparing performance vectors.
static java.lang.String	PARAMETER_SKIP_UNDEFINED_LABELS The parameter name for "If set to true, examples with undefined labels are skipped.

Constructor Summary

AbstractPerformanceEvaluator(OperatorDescription description)

Method Summary

IOObject[]	apply() Implement this method in subclasses.
protected abstract void	checkCompatibility(ExampleSet exampleSet) Performs a check if this operator can be used for this type of exampel set at all.
static void	evaluate(AbstractPerformanceEvaluator evaluator, ExampleSet testSet, PerformanceVector performanceCriteria, java.util.List<PerformanceCriterion> givenCriteria, boolean skipUndefinedLabels, boolean useExampleWeights) Static version of evaluate(ExampleSet,PerformanceVector).
protected PerformanceVector	evaluate(ExampleSet testSet, PerformanceVector inputPerformance) Evaluates the given test set.
protected abstract double[]	getClassWeights(Attribute label) Delivers class weights for performance criteria which implement the ClassWeightedPerformance interface.
abstract java.util.List<PerformanceCriterion>	getCriteria() Delivers the list of criteria which is able for this operator.
java.lang.Class<?>[]	getInputClasses() Returns the classes that are needed as input.
InputDescription	getInputDescription(java.lang.Class cls) Shows a parameter keep_example_set with default value "false".
java.lang.Class<?>[]	getOutputClasses() Returns the classes that are guaranteed to be returned by apply() as additional output.
java.util.List<ParameterType>	getParameterTypes() Returns a list of ParameterTypes describing the parameters of this operator.
protected void	init(ExampleSet exampleSet) This method will be invoked before the actual calculation is started.
protected boolean	showComparatorParameter() Subclasses might override this method and return false.
protected boolean	showCriteriaParameter() Subclasses might override this method and return false.
protected boolean	showSkipNaNLabelsParameter() Subclasses might override this method and return false.

Methods inherited from class com.rapidminer.operator.Operator

addError, addValue, addWarning, apply, checkDeprecations, checkForStop, checkIO, checkProperties, clearErrorList, cloneOperator, createExperimentTree, createExperimentTree, createFromXML, createMarkedExperimentTree, createMarkedProcessTree, createProcessTree, createProcessTree, getAddOnlyAdditionalOutput, getApplyCount, getDeliveredOutputClasses, getDeprecationInfo, getDesiredInputClasses, getEncoding, getErrorList, getExperiment, getInnerOperatorsXML, getInput, getInput, getInput, getIOContainerForInApplyLoopBreakpoint, getIODescription, getLog, getName, getOperatorClassName, getOperatorDescription, getParameter, getParameterAsBoolean, getParameterAsColor, getParameterAsDouble, getParameterAsFile, getParameterAsFile, getParameterAsInputStream, getParameterAsInt, getParameterAsMatrix, getParameterAsString, getParameterList, getParameters, getParameterType, getParent, getProcess, getStartTime, getStatus, getUserDescription, getValue, getValues, getXML, hasBreakpoint, hasBreakpoint, hasInput, inApplyLoop, isDebugMode, isEnabled, isExpanded, isParallel, isParameterSet, log, logError, logNote, logWarning, performAdditionalChecks, processFinished, processStarts, register, registerOperator, remove, rename, resume, setApplyCount, setBreakpoint, setEnabled, setExpanded, setInput, setListParameter, setOperatorParameters, setParameter, setParameters, setParent, setUserDescription, toString, unregisterOperator, writeXML

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

PARAMETER_MAIN_CRITERION

public static final java.lang.String PARAMETER_MAIN_CRITERION

The parameter name for "The criterion used for comparing performance vectors."

See Also:

Constant Field Values

PARAMETER_SKIP_UNDEFINED_LABELS

public static final java.lang.String PARAMETER_SKIP_UNDEFINED_LABELS

The parameter name for "If set to true, examples with undefined labels are skipped."

See Also:

Constant Field Values

PARAMETER_COMPARATOR_CLASS

public static final java.lang.String PARAMETER_COMPARATOR_CLASS

The parameter name for "Fully qualified classname of the PerformanceComparator implementation."

See Also:

Constant Field Values

Constructor Detail

AbstractPerformanceEvaluator

public AbstractPerformanceEvaluator(OperatorDescription description)

Method Detail

getCriteria

public abstract java.util.List<PerformanceCriterion> getCriteria()

Delivers the list of criteria which is able for this operator. Please note that all criteria in the list must be freshly instantiated since no copy is created in different runs of this operator. This is important in order to not mess up the results. This method must not return null but should return an empty list in this case.

getClassWeights

protected abstract double[] getClassWeights(Attribute label)
                                     throws UndefinedParameterError

Delivers class weights for performance criteria which implement the ClassWeightedPerformance interface. Might return null (for example for regression task performance evaluators).

Throws:

UndefinedParameterError

checkCompatibility

protected abstract void checkCompatibility(ExampleSet exampleSet)
                                    throws OperatorException

Performs a check if this operator can be used for this type of exampel set at all.

Throws:

OperatorException

init

protected void init(ExampleSet exampleSet)

This method will be invoked before the actual calculation is started. The default implementation does nothing. Subclasses might want to override this method.

showSkipNaNLabelsParameter

protected boolean showSkipNaNLabelsParameter()

Subclasses might override this method and return false.

showComparatorParameter

protected boolean showComparatorParameter()

Subclasses might override this method and return false.

showCriteriaParameter

protected boolean showCriteriaParameter()

Subclasses might override this method and return false.

apply

public IOObject[] apply()
                 throws OperatorException

Description copied from class: Operator

Implement this method in subclasses.

Specified by:

apply in class Operator

Throws:

OperatorException

evaluate

protected PerformanceVector evaluate(ExampleSet testSet,
                                     PerformanceVector inputPerformance)
                              throws OperatorException

Evaluates the given test set. All PerformanceCriterion instances in the given PerformanceVector must be subclasses of MeasuredPerformance.

Throws:

OperatorException

evaluate

public static void evaluate(AbstractPerformanceEvaluator evaluator,
                            ExampleSet testSet,
                            PerformanceVector performanceCriteria,
                            java.util.List<PerformanceCriterion> givenCriteria,
                            boolean skipUndefinedLabels,
                            boolean useExampleWeights)
                     throws OperatorException

Static version of evaluate(ExampleSet,PerformanceVector). This method was introduced to enable testing of the method.

Parameters:

evaluator - Ususally this. May be null for testing. Only needed for exception.

Throws:

OperatorException

getInputDescription

public InputDescription getInputDescription(java.lang.Class cls)

Shows a parameter keep_example_set with default value "false".

Overrides:

getInputDescription in class Operator

getInputClasses

public java.lang.Class<?>[] getInputClasses()

Description copied from class: Operator

Returns the classes that are needed as input. May be null or an empty (no desired input). As default, all delivered input objects are consumed and must be also delivered as output in both Operator.getOutputClasses() and Operator.apply() if this is necessary. This default behavior can be changed by overriding Operator.getInputDescription(Class). Subclasses which implement this method should not make use of parameters since this method is invoked by getParameterTypes(). Therefore, parameters are not fully available at this point of time and this might lead to exceptions. Please use InputDescriptions instead.

Specified by:

getInputClasses in class Operator

getOutputClasses

public java.lang.Class<?>[] getOutputClasses()

Description copied from class: Operator

Returns the classes that are guaranteed to be returned by apply() as additional output. Please note that input objects which should not be consumed must also be defined by this method (e.g. an example set which is changed but not consumed in the case of a preprocessing operator must be defined in both, the methods Operator.getInputClasses() and Operator.getOutputClasses()). The default behavior for input consumation is defined by Operator.getInputDescription(Class) and can be changed by overwriting this method. Objects which are not consumed (defined by changing the implementation in Operator.getInputDescription(Class)) must not be defined as additional output in this method.

May deliver null or an empy array (no additional output is produced or guaranteed). Must return the class array of delivered output objects otherwise.

Specified by:

getOutputClasses in class Operator

getParameterTypes

public java.util.List<ParameterType> getParameterTypes()

Description copied from class: Operator

Returns a list of ParameterTypes describing the parameters of this operator. The default implementation returns an empty list if no input objects can be retained and special parameters for those input objects which can be prevented from being consumed.

Specified by:

getParameterTypes in interface ParameterHandler

Overrides:

getParameterTypes in class Operator