Pages: [1]
  Print  
Author Topic: confidences and probability  (Read 2252 times)
babaluma
Newbie
*
Posts: 3


« on: July 27, 2008, 03:50:05 AM »

I am trying to feed RapidMiner output (predictions and probabilities)  into an external program. I understand how to output confidence scores using ExampleSetWriter, but not probabilities. I found a thread in the old SourceForge forum that suggests using PlattScaling to convert confidence scores to probabilities. I looked around a little bit, but could not get this operator to get what I want. How should I go about doing this? How do I find and write the probability that a particular instance belongs to a class?

Thanks,
B
Logged
steffen
Sr. Member
****
Posts: 376



« Reply #1 on: July 28, 2008, 09:25:10 AM »

Hello babaluma

For others: Here is the link to the mentioned sf-thread: click.

Regarding your problem:
I understand Platt Scaling this way:
input: the ExampleSet the original model was created with and the created model
output: the "corrected" model.

Applying the corrected model to your test set results in real probabilities in the confidence column, i.e. confidence(c)=p(y=c|x=current Example).

hope this was helpful

Steffen

PS: This topic is really interesting. Recently I stumbled upon a paper converting scores of multiple-class-classifiers, but unfortunately, I did not have the time to study it yet. Here is the link (PDF):Transforming Classifier Scores into Accurate Multiclass Probability Estimates
Logged

"I want to make computers do what I mean instead of what I say"
Read The Fantastic Manual
babaluma
Newbie
*
Posts: 3


« Reply #2 on: July 28, 2008, 08:53:48 PM »

thanks for the reply.

while that seems to work, I am having trouble in getting probabilities from the training set with cross-validation. Where exactly should PlattScaling operator be placed in such a scenario?
My cross validation step looks like the following:
-XValidation
--NaiveBayes
--OperatorChain
---ModelApplier
---BinomialClassificationPerformance
---ExampleWriter
---ResultWriter
-ModelWriter
-PerformanceWriter

It seems that PlattScaling should be performed after the learning step and before the ExampleWriter operator. However, by placing it before the ExampleWriter operator, ExampleSet is lost (PlattScaling takes ExampleSet and Model and returns Model). Placing it after the learning step, I somehow get an error in the PerformanceVector operator. The error is that "an object of PerformanceVector type is needed". Without PlattScaling, it works fine. I suppose if PlattScaling had the parameter "keep_example_set", this would work. I would appreciate any feedback.

Thanks
B


 
Logged
steffen
Sr. Member
****
Posts: 376



« Reply #3 on: July 28, 2008, 09:26:31 PM »

Hello again

If you want to create a "platt-scaled" model for each training step, I suggest something like this: (simple copy and paste it to the xml-tab of rm-interface)
Code:
<operator name="Root" class="Process" expanded="yes">
    <operator name="XValidation" class="XValidation" expanded="yes">
        <operator name="create_scaled_model" class="OperatorChain" expanded="yes">
            <operator name="NaiveBayes" class="NaiveBayes">
                <parameter key="keep_example_set" value="true"/>
            </operator>
            <operator name="PlattScaling" class="PlattScaling">
            </operator>
        </operator>
        <operator name="OperatorChain" class="OperatorChain" expanded="yes">
            <operator name="ModelApplier" class="ModelApplier">
                <list key="application_parameters">
                </list>
            </operator>
            <operator name="BinominalClassificationPerformance" class="BinominalClassificationPerformance">
            </operator>
        </operator>
    </operator>
</operator>

Hope this is what you wanted to do Smiley

greetings

Steffen
« Last Edit: July 28, 2008, 09:28:23 PM by steffen » Logged

"I want to make computers do what I mean instead of what I say"
Read The Fantastic Manual
babaluma
Newbie
*
Posts: 3


« Reply #4 on: July 30, 2008, 04:23:45 AM »

That seems to work. Thanks a lot.

B
Logged
Pages: [1]
  Print  
 
Jump to: