Pages: [1]
  Print  
Author Topic: SOM Reduction - Preprocessing Save doesn't work  (Read 3456 times)
BCOURTNEY
Guest
« on: June 17, 2008, 10:17:44 PM »

When I attempt to create and save a preprocessing model with 205 features ( a mix of nominal and numeric features ) with about 10,000 training rows, after processing for about a hour, the modelwrite writes out about a 12 meg model, then hangs and freezes. Any attempts to use this model results in errors. Is this a size or complexity limitation?
Logged
Tobias Malbrecht
Global Moderator
Sr. Member
*****
Posts: 293



WWW
« Reply #1 on: June 17, 2008, 10:45:46 PM »

Hi,

which preprocessing model did you create and save? Could you please post your process XML?! Given your description, I have no idea what could be the problem here.

Regards,
Tobias
Logged

Tobias Malbrecht
Director of Product Marketing
RapidMiner
Sebastian Land
Administrator
Hero Member
*****
Posts: 2426


« Reply #2 on: June 17, 2008, 11:55:32 PM »

Hi,
since the SOMReduction should save the generated SOM, there should be no complexity problem with the number of examples. (Ignoring time needed to compute the SOM)
How long did it freeze, before it terminated the process? It might be, that saving might last a few minutes.

A quoting of the error messages would be usefull, too.


Greetings,
   Sebastian
Logged
BCOURTNEY
Guest
« Reply #3 on: June 18, 2008, 03:40:44 AM »

Here's the error message when trying to load the model created from the SOM preprocessing

Exception: com.rapidminer.operator.UserError
Message: Could not read file 'F:\ModelDevelopment\preprocess_som.mod': Cannot read from XML stream, wrong format: Could not call java.security.CodeSource.readObject()
---- Debugging information ----
message             : Could not call java.security.CodeSource.readObject()
cause-exception     : java.lang.RuntimeException
cause-message       : null
class               : com.rapidminer.operator.ContainerModel
required-type       : java.security.CodeSource
line number         : 2224478
-------------------------------.
Stack trace:

  com.rapidminer.operator.io.ModelLoader.apply(ModelLoader.java:97)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.Process.run(Process.java:571)
  com.rapidminer.Process.run(Process.java:541)
  com.rapidminer.Process.run(Process.java:531)
  com.rapidminer.gui.ProcessThread.run(ProcessThread.java:61)

Cause
Exception: java.io.StreamCorruptedException
Message: invalid stream header: 1F8B0800
Stack trace:

  java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
  java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
  com.rapidminer.operator.io.ModelLoader.apply(ModelLoader.java:76)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.operator.OperatorChain.apply(OperatorChain.java:377)
  com.rapidminer.operator.Operator.apply(Operator.java:656)
  com.rapidminer.Process.run(Process.java:571)
  com.rapidminer.Process.run(Process.java:541)
  com.rapidminer.Process.run(Process.java:531)
  com.rapidminer.gui.ProcessThread.run(ProcessThread.java:61)
Logged
Sebastian Land
Administrator
Hero Member
*****
Posts: 2426


« Reply #4 on: June 18, 2008, 03:27:18 PM »

Hi,
I just found your bug. Funny one Smiley Since the SOM was a thread itself, the IOObjectWriter tried to save nearly everything. Complete rapidMiner, libraries, everything, since the thread knew its threadpool, which probably knew all threads including the gui...And when the IOObjectWriter started to save itself, it hang up. What was probably a good idea after all Smiley

In the newest repository version it now works.

Greetings,
   Sebastian
Logged
bcourtney
Guest
« Reply #5 on: June 19, 2008, 02:10:29 AM »

Great. How or where do I access the repository version? I assume it's CVS somewhere, and is the repository built into binaries or do I have to compile ... ?
Logged
Tobias Malbrecht
Global Moderator
Sr. Member
*****
Posts: 293



WWW
« Reply #6 on: June 19, 2008, 08:36:59 AM »

Hi,

http://rapid-i.com/content/view/25/48/lang,de/

explains the anonymous CVS access to our developer version via Eclipse. The developer CVS repository naturally only contains the source code. The compilation or running of RapidMiner, respectively, is pretty easy using Eclipse. Just follow the instructions which are explained on the aforementioned website.

Regards,
Tobias
Logged

Tobias Malbrecht
Director of Product Marketing
RapidMiner
bcourtney
Guest
« Reply #7 on: June 20, 2008, 04:14:53 AM »

Checked it out with eclipse. Compile worked great, including the fix on the SOM preprocessing saving/loading issue. Thanks.
Logged
Pages: [1]
  Print  
 
Jump to: