Pages: [1]
  Print  
Author Topic: Excessive memory usage: rapid-4-1-beta vs rapid-4-1  (Read 1754 times)
meb
Newbie
*
Posts: 1


« on: June 19, 2008, 08:40:59 PM »

I was wondering whether anyone else shared my experience with the latest community edition rapidminer-4.1?

For a while I was using rapidminer-4.1-beta. Then, I installed the update rapidminer-4.1 and memory usage seems excessive. My whole machine slows down and the hard drive light on solid. Task manager shows a lot of page faulting going on.

I'm using Windows XP (actually, XP is using me).

I'm considering going back to the beta version. That didn't give me problems like this.
Logged
steffen
Sr. Member
****
Posts: 376



« Reply #1 on: June 20, 2008, 08:46:35 AM »

Hello

I am also using Windows XP on a P4 with one cpu 3.06 GHz and 1 GB RAM. In my point of view, the memory usage of RapidMiner 4.1 is much better than in earlier versions. I create a lot of "iterating processes" , which stuck in earlier versions 'cause RM was not able to remove unused result objects from the memory properly (between two iteration steps). But now, the aforesaid processes run without problems.

Which kind of computer do you use ?

greetings

Steffen
Logged

"I want to make computers do what I mean instead of what I say"
Read The Fantastic Manual
Tobias Malbrecht
Global Moderator
Sr. Member
*****
Posts: 293



WWW
« Reply #2 on: June 20, 2008, 01:47:49 PM »

Hi,

I also have not the impression, that RapidMiner 4.1 is gravely consuming more memory than the previous beta version. Moreover that would be surprising since the work we currently do and have done for the last weeks is not only dedicated to the invention and implementation of new functionality for RapidMiner, but to a large extent also to the improvement of existing functions, operators, etc. due to further improve the performance and scalablity of RapidMiner. Nevertheless changes in the implementation might in some cases have side-effects such as higher memory consumption, which is unintentional in some cases or accepted in favour of other advantages (useability, functionality, ect.). However, we greatly appreciate users who report cases in which the usage of particular operators or processes and who reflect the need to either communicate some changes more clearly or point at things that can be potentially be improved.

To cut a long story short, if you experience significant losses of performance (i.e. also increased memory consumption) in particular processes or operators, please let us know. Unfortunately, a general or global statement such as yours is otherwise relatively hard to retrace.

Regards,
Tobias

Logged

Tobias Malbrecht
Director of Product Marketing
RapidMiner
Ingo Mierswa
Administrator
Hero Member
*****
Posts: 1226



WWW
« Reply #3 on: June 24, 2008, 07:07:06 PM »

Hello,

we made some tests on a set of 143 different data mining processes and the version 4.1 is using 22% less memory than before on average over all processes and there was no process which used more memory. Actually, a lot of work of the last months was put in the optimization with respect to memory consumption and / or runtimes (but of course a lot is still to do).

However, could it be that you changed to the 64 bit version? This version uses the double amount of memory for all internal references and hence double memory usage. So the 64 bit version is only recommended for systems with more than approx. 3,5 Gb system memory.

Cheers,
Ingo
Logged

Did you try our new Marketplace? Upload or download new Extensions, add comments, and organize your operators. Have a look at  http://marketplace.rapid-i.com
Pages: [1]
  Print  
 
Jump to: