Pages: [1]
  Print  
Author Topic: Only minimum data can handled by k-means  (Read 715 times)
venkat
Newbie
*
Posts: 8


« on: April 09, 2014, 11:37:56 AM »

Hi All,

While trying with k-means algorithm, Rapidminer can able to process only 335mb not more that. 

here is my XML

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="read_csv" compatibility="5.3.013" expanded="true" height="60" name="Read CSV" width="90" x="45" y="75">
        <parameter key="csv_file" value="/root/Desktop/rapidsimpletest/337mb.csv"/>
        <list key="annotations"/>
        <list key="data_set_meta_data_information"/>
      </operator>
      <operator activated="true" class="normalize" compatibility="5.3.013" expanded="true" height="94" name="Normalize" width="90" x="246" y="255"/>
      <operator activated="true" class="k_means" compatibility="5.3.013" expanded="true" height="76" name="Clustering" width="90" x="447" y="120">
        <parameter key="k" value="3"/>
        <parameter key="measure_types" value="MixedMeasures"/>
      </operator>
      <connect from_op="Read CSV" from_port="output" to_op="Normalize" to_port="example set input"/>
      <connect from_op="Normalize" from_port="example set output" to_op="Clustering" to_port="example set"/>
      <connect from_op="Clustering" from_port="cluster model" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>


Erro Description:
Apr 09, 2014 4:01:53 PM com.rapidminer.tools.WrapperLoggingHandler log
WARNING: Read CSV: Could not parse line 24 in input: com.rapidminer.tools.CSVParseException: Value quote misplaced at position 88. Last characters read: ormation,"
Apr 09, 2014 4:01:53 PM com.rapidminer.tools.WrapperLoggingHandler log
WARNING: Read CSV: Could not parse line 24 in input: com.rapidminer.tools.CSVParseException: Value quote misplaced at position 42. Last characters read:  Archive,"
Apr 09, 2014 4:01:53 PM com.rapidminer.tools.WrapperLoggingHandler log
WARNING: Read CSV: Could not parse line 24 in input: com.rapidminer.tools.CSVParseException: Value quote misplaced at position 67. Last characters read:  Home,,,,"
Apr 09, 2014 4:01:53 PM com.rapidminer.tools.WrapperLoggingHandler log
WARNING: Read CSV: Maximum number of warnings exceeded. Will display no further warnings.
Apr 09, 2014 4:02:06 PM com.rapidminer.gui.ProcessThread run
SEVERE: Process failed: Java heap space
java.lang.OutOfMemoryError: Java heap space
   at java.util.Arrays.copyOf(Arrays.java:2367)
   at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
   at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
   at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:535)
   at java.lang.StringBuffer.append(StringBuffer.java:322)
   at java.io.BufferedReader.readLine(BufferedReader.java:351)
   at java.io.BufferedReader.readLine(BufferedReader.java:382)
   at com.rapidminer.gui.tools.dialogs.wizards.dataimport.csv.LineReader.readLine(LineReader.java:55)
   at com.rapidminer.operator.nio.model.CSVResultSet.readNext(CSVResultSet.java:149)
   at com.rapidminer.operator.nio.model.CSVResultSet.next(CSVResultSet.java:195)
   at com.rapidminer.operator.nio.model.DataResultSetTranslator.read(DataResultSetTranslator.java:148)
   at com.rapidminer.operator.nio.model.AbstractDataResultSetReader.createExampleSet(AbstractDataResultSetReader.java:147)
   at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:52)
   at com.rapidminer.operator.io.AbstractExampleSource.read(AbstractExampleSource.java:36)
   at com.rapidminer.operator.io.AbstractReader.doWork(AbstractReader.java:126)
   at com.rapidminer.operator.Operator.execute(Operator.java:867)
   at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
   at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
   at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:375)
   at com.rapidminer.operator.Operator.execute(Operator.java:867)
   at com.rapidminer.Process.run(Process.java:949)
   at com.rapidminer.Process.run(Process.java:873)
   at com.rapidminer.Process.run(Process.java:832)
   at com.rapidminer.Process.run(Process.java:827)
   at com.rapidminer.Process.run(Process.java:817)
   at com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)

Apr 09, 2014 4:02:06 PM com.rapidminer.gui.ProcessThread run
SEVERE: Here:           Process[1] (Process)
           subprocess 'Main Process'
       ==>   +- Read CSV[1] (Read CSV)
             +- Normalize[0] (Normalize)
             +- Clustering[0] (k-Means)


I am trying 3 clusters with k-means. But the results are not proper.

Cluster Model
Cluster 0: 22292 items
Cluster 1: 1 items
Cluster 2: 1 items
Total number of items: 22294

could you please guys help me, where I am doing mistakes?

Thanks.
Venkat


Logged
Marius
Administrator
Hero Member
*****
Posts: 1794



WWW
« Reply #1 on: April 10, 2014, 09:58:16 AM »

How much main memory did you assign to RapidMiner?
How many rows and columns do your datasets have?

Best regards,
Marius
Logged

Please add [SOLVED] to the topic title when your problem has been solved! (do so by editing the first post in the thread and modifying the title)
Please click here before posting.
venkat
Newbie
*
Posts: 8


« Reply #2 on: April 10, 2014, 10:10:44 AM »

Hi Marius,

I have given 1 GB for main memory. My input file contains 143000 rows and 5 columns. The size of my input file is 337 MB.

I can able to process the 335MB.

Reg,
Venkat
Logged
Marius
Administrator
Hero Member
*****
Posts: 1794



WWW
« Reply #3 on: April 10, 2014, 10:24:48 AM »

1 GB is not much for a data mining process. RapidMiner often expands the read data in main memory to optimize it for the data mining tasks. The algorithms then oftentimes also need more memory than just the size of the pure data to create the models etc.

You should definitely try to increase the amount of avilable memory.

Best regards,
Marius
Logged

Please add [SOLVED] to the topic title when your problem has been solved! (do so by editing the first post in the thread and modifying the title)
Please click here before posting.
Pages: [1]
  Print  
 
Jump to: