This year's RCOMM live data mining challenge, "Who wants to be a data miner?", was a tricky yet fun task for the competitors. The task was to (partially) solve a Sudoku puzzle with RapidMiner. The solution shows that you can achieve virtually any data analysis task you can think of only by using standard RapidMiner processes.

The input data set consisted of examples of the form (x,y,v) where x and y indicated the column and row inside the puzzle and v was the number predefined at this cell. The path to the solution was split into three subtasks:

1. Task one was to generate the space of all combinations of cells and numbers that were possible if there were no predefined numbers in the Sudoku.This task could be solved by starting with a simple data set only containing the numbers one to nine and applying two Cartesian Product operators to generate all combinations of these numbers for x, y, and v. In addition to that, we also generate a new attribute z, which indicates the 3x3-sub-table in which the cell (x,y) lies, using a Generate Attributes operator.
   
2. This additional attribute z is useful for Subtask 2 which was to eliminate all combinations which are impossible, given a single predefined cell value from the input data set. This was possible by using a combination of a Generate Attributes operator and a Filter Examples operator to identify those combinations (from the set of all combinations generated in Subtask 1) where v and at least one of x, y, or z match a number defined in the input.
3. The resulting process of Subtask 2 could be re-used in Subtask 3 to eliminate all combinations whose impossibility could be inferred from looking at all predefined cell values in the input data set. This could be achieved by using a Loop Examples operator to iterate over all cells and using a nested Execute Process to re-use the process generated in Subtask 2.
   Finally, by looking at those cells where only one possible number remains,we can identify a new value that can be inserted into the Sudoku for sure. These cells can be identified by using an Aggregate operator to group the remaining possibilities by x, y, and z and find those groups with a count of exactly one.
4. As a bonus process, we can repeat the process from Subtask 3, iteratively Appending the inferred numbers to the predefined ones. Thus, the data set grows up to a size of 81 which means the 9x9 Sudoku is complete. Finally, we use the Pivot operator and do some polishing to make the result look like this, a completely solved Sudoku:
   

All the processes mentioned above are accessible from myExperiment as a pack which you can use from within RapidMiner by using the Community Extension. The processes can be directly opened in RapidMiner, but when saving make sure to name them as they are named in myExperiment since an Execute Process operator in a later process may expect it under this name. Process 0 in this pack downloads the initial data from the Web.

A new data mining challenge has been launched within the e-LICO project.

The challenge is called ECML/PKDD Discovery Challenge 2011: VideoLectures.Net recommender system. The contest consists of two recommender system problem tasks and a side-by workflow contest in which Rapid-I sponsors the best RapidMiner workflow with an free admission to RCOMM 2012 including flight and hotel.

The total award of the challenge sums to Euro 5500 - plus the free trip to the RCOMM 2012!

All details on the challenge can be found at

http://www.ecmlpkdd2011.org/challenge.php.

We are looking forward to your RapidMiner workflows.

Good luck!

Within the e-LICO project, Rapid-I is sponsoring a data mining challenge on Obstructive Nephropathy (ON). The task is particularly challenging since it exhibits the typical charecteristics of data from the bio domain: high dimensionality versus small sample size, incomplete data, and a high degree of dependencies. The challenge is open to all participants now and the award includes a prize money of Euro 2500.

We encourage all users of RapidMiner to participate in this challenge - chances are high to come up with a good solution! The challenge ends on December 19th, 2010. Please find more information on the challenge web page at

At the RCOMM, we had a challenge in which data miners had to design RapidMiner processes solving unusual tasks. The three tasks were to design a process that creates the lyrics of "99 bottles of beer", apply a model on a data set of which a complete column was lost, and to create a process that computes the Fibonacci numbers. All winning solutions, challenge descriptions, and necessary data preparation processes are now on myExperiment:

http://www.myexperiment.org/search?query=rcomm&type=all

I think they are worth looking at since they apply quite some clever tricks.

Furthermore, we have seen a lot of interesting and brand-new RapidMiner Extensions at the conference. One of them, made by the DFKI, assists the data miner in choosing an appropriate learner for their data set and saves you from trying a lot of different learners manually. The extensions is available from our update server and is described here:

 http://madm.dfki.de/rapidminer/wizard

Try it out!