Pages: [1]
  Print  
Author Topic: Maximum size of input  (Read 2106 times)
fervlrm
Newbie
*
Posts: 2


« on: June 27, 2008, 01:05:25 PM »

Hi all,

I am learning on rapid miner but I would like to know if it will be able to handle a source CSV file with 30 million entries, containing each 26 attributes.... Can rapidminer handle it?

Thanks
Logged
fervlrm
Newbie
*
Posts: 2


« Reply #1 on: June 27, 2008, 01:32:29 PM »

In fact,

I have tried to use the ExampleSetGenerator to generate 27.000.000 samples with 26 attributes and it says JavaHeap Memory error.....
any solution?
Logged
vps
Newbie
*
Posts: 30


« Reply #2 on: June 27, 2008, 07:43:30 PM »

Hi,
Simple Solution: Use 64 bit machine and increase the RAM memory..

I know matlab and IDL have file association with variable that allows to read only the required part of the file, I am not sure if Java supports it. May be you might want to research on that.

Regards,
Vijay
Logged
Ingo Mierswa
Administrator
Hero Member
*****
Posts: 1226



WWW
« Reply #3 on: June 27, 2008, 10:12:42 PM »

Hi,

yes, increasing the available memory is certainly an option. Another option is to store the data in a database and directly work on it with the appropriate settings.

Cheers,
Ingo
Logged

Did you try our new Marketplace? Upload or download new Extensions, add comments, and organize your operators. Have a look at  http://marketplace.rapid-i.com
Pages: [1]
  Print  
 
Jump to: