Pages: [1]
  Print  
Author Topic: Lasso  (Read 2887 times)
Ben
Newbie
*
Posts: 24


WWW
« on: March 31, 2010, 10:52:19 AM »

Hi,
I was wondering whether there exist any Lasso-implementations in RM? (least-angle-regression) Or am I just too blind to find one?

Greetings
Ben
« Last Edit: April 12, 2010, 11:23:37 AM by Ben » Logged

http://www-ai.cs.tu-dortmund.de/PERSONAL/schowe.html
Artificial Intelligence Group
Technical University of Dortmund
Sebastian Land
Administrator
Hero Member
*****
Posts: 2426


« Reply #1 on: April 12, 2010, 10:50:44 AM »

Hi Ben,
it's currently not included, but we are planning to implement it in near future. The problem is that the optimization is not that easy as for example the ridge regression or simple linear regression: It does not collapse to a closed form and hence needs some more complex iterative optimization. If you have a suggestion or implementation at hand, feel free to contribute Smiley

Greetings,
  Sebastian
Logged
Ben
Newbie
*
Posts: 24


WWW
« Reply #2 on: December 02, 2010, 10:15:51 PM »

FYI, I have now implemented LARS (Least Angle Regression). I will include LASSO-modifications the next days and will release the operator approximately next week in the new version of my Microarray Feature Selection Plugin.

Ben
Logged

http://www-ai.cs.tu-dortmund.de/PERSONAL/schowe.html
Artificial Intelligence Group
Technical University of Dortmund
Ben
Newbie
*
Posts: 24


WWW
« Reply #3 on: December 08, 2010, 03:13:47 PM »

And there it is:
Least Angle Regression (LARS) and LASSO ready to serve you at:

https://sourceforge.net/projects/rm-featselext/

It's a pretty straight-forward implementation of the LARS-algorithm described in Least Angle Regression by Efron et al. 2004 in "The Annals of Statistics".
It works so far, but I have not (yet) included the speed-optimization described in a later section of the paper.

Enjoy,
Ben
Logged

http://www-ai.cs.tu-dortmund.de/PERSONAL/schowe.html
Artificial Intelligence Group
Technical University of Dortmund
dragoljub
Full Member
***
Posts: 246


« Reply #4 on: December 10, 2010, 06:31:48 PM »

Sweet thanks for the great work!

-Gagi  Grin
Logged
wessel
Hero Member
*****
Posts: 558


« Reply #5 on: December 10, 2010, 06:47:10 PM »

Cool.

I also really like the added screen shots.

Logged
dragoljub
Full Member
***
Posts: 246


« Reply #6 on: December 15, 2010, 07:31:08 PM »

Your plug-in looks great. I am considering using your RFE operator, however I am not 100% sure how it differs from the optimize weights using backward elimination.

It seems that RFE reduces the selected features by removing the least weighted features and re-weighting. Backward elimination seems to do the same thing (drop worst features) however is more tied to performance of a specific model and will likely blow up since many combination of dropped features must be tried.

Any input is appreciated.  Grin

Thanks,
-Gagi
Logged
dragoljub
Full Member
***
Posts: 246


« Reply #7 on: December 16, 2010, 07:05:23 AM »

I have another question. Does the RFE operator also handle +/- feature weights. For example the SVM operator will provide the actual +/- weights where as the weight by SVM operator will provide normalized weights. Do we need normalized weights to use RFE?  Cheesy

Thanks,
-Gagi
Logged
Ben
Newbie
*
Posts: 24


WWW
« Reply #8 on: December 20, 2010, 12:19:48 PM »

I had to check on this :-)

So for versions =< 1.0.3 the following holds

The SVM-RFE operator uses the absolute weights of the inner SVM.
In detail: An invisible SVM-Weighting-operator calculates the weights. They may contain negative values.
Then the weights are normalized. The normalization-method maps the absolute values to the range [0..1]. That means negative weights with a large absolute value will a high value  (-> 1) after normalization. see AttributeWeights.normalize() for details.

The RFE operator with arbitrary subprocess does not (yet) use absolute values. This means that if you want negative weights with a high absolute value to remain in the selection, you must normalize them or convert them to absolute values.

 Wait a second.
...
I added a parameter "use_absolute_weights" to the RFE-operator. It is availalbe in the SVN version (trunk) but not yet released as prebuilt JAR.



Regarding the question of your previous post RFE vs. Backward-Elimination: They are quite different! :-)
First, the similarity. BE and RFE both remove features from the full set of features, BUT:
In every round BE has to evaluate the prediction performance of removing each feature by cross-validation or bootstrap or the like. That's very expensive.
RFE in contrast has only to calculate one weight-vector per round and the removes the feature(s)* with the smallest weight.
The RFE operator usually removes several attributes in one round whereas the BE-operator only removes one feature each round (I'm not 100% sure about that).


Merry Christmas,
Ben
Logged

http://www-ai.cs.tu-dortmund.de/PERSONAL/schowe.html
Artificial Intelligence Group
Technical University of Dortmund
dragoljub
Full Member
***
Posts: 246


« Reply #9 on: December 25, 2010, 03:25:30 AM »

Thanks Ben!

I will have to get the latest version once you build the jar. For now I will use the weight by SVM for normalized weights.

Thanks and Merry Christmas,
-Gagi
Logged
dragoljub
Full Member
***
Posts: 246


« Reply #10 on: December 27, 2010, 11:29:48 PM »

Hey Ben,

Is it possible to return the non normalized weights at the end of your operator. Say I want the 10 best features after RFE but I also want to know the relative ranking of these 10 features. It would be great to output the actual weights of the final iteration rather than just 0/1.  Grin

Thanks,
-Gagi
Logged
Ben
Newbie
*
Posts: 24


WWW
« Reply #11 on: January 03, 2011, 11:47:19 AM »

Hi!

I'm sorry, but that's not possible at the moment and would interfere with the purpose of feature selection. I think you have to apply another round of SVM-Weighting on the selected features.

I'll let you know, if I find a workaround for that.

Ben
Logged

http://www-ai.cs.tu-dortmund.de/PERSONAL/schowe.html
Artificial Intelligence Group
Technical University of Dortmund
dragoljub
Full Member
***
Posts: 246


« Reply #12 on: January 04, 2011, 01:04:34 AM »

OH OK that makes sense. I will give that a try.

Thanks Ben!
-Gagi
Logged
Pages: [1]
  Print  
 
Jump to: