Pages: [1]
  Print  
Author Topic: Problem with R-script results  (Read 371 times)
ighyboo
Newbie
*
Posts: 26


« on: October 02, 2014, 12:41:36 PM »

Hi everyone,
I used the R extension before and always managed to get the data out using "outdata <-as.data.frame(Results)"

I'm currently working on a script that download twitter followers using the twitter API, everything work fine and I managed to create the results data-frame but Rapidminer doesn't like it and spit out this error:

"In order to import an R Data Frame as example set the data frame must provide attribute names"

I'm confused because when I check my result variable with colnames(outdata), the column names are assigned as you can see here:

colnames(Results)
 [1] "id_str"            "name"              "screen_name"     
 [4] "url"               "profile_image_url" "description"     
 [7] "location"          "followers_count"   "friends_count"   
[10] "statuses_count"    "created_at"   



Anyone else had this problem before?

Here's the process and the script:
Code:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.015">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.015" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data_user_specification" compatibility="5.3.015" expanded="true" height="60" name="Generate Data by User Specification" width="90" x="45" y="30">
        <list key="attribute_values">
          <parameter key="target" value="&quot;WRADARltd&quot;"/>
        </list>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="r:execute_script_r" compatibility="5.3.000" expanded="true" height="76" name="Execute Script (R)" width="90" x="179" y="30">
        <parameter key="script" value="#Install libraries&#10;install.packages(&quot;httr&quot;);&#10;install.packages(&quot;rjson&quot;);&#10;install.packages(&quot;data.table&quot;);&#10;library(httr);&#10;library(rjson);&#10;library(data.table);&#10;&#10;&#10;#Declare API keys and inputs&#10;consumer_key = &quot;UansBjFHOm8fpkB4jx5mSiDTu&quot;;&#10;consumer_secret = &quot;dKTbQ0QKCqHaejnLyHOW0PHuOzpn5PIhjVmOkGMzcyLuaFrA7p&quot;;&#10;target = indata$target;&#10;&#10;&#10;#Auth&#10;secret &lt;- RCurl::base64(paste(consumer_key, consumer_secret, sep = &quot;:&quot;));&#10;req &lt;- POST(&quot;https://api.twitter.com/oauth2/token&quot;,&#10;  config(httpheader = c(&#10;    &quot;Authorization&quot; = paste(&quot;Basic&quot;, secret),&#10;    &quot;Content-Type&quot; = &quot;application/x-www-form-urlencoded;charset=UTF-8&quot;&#10;  )),&#10;  body = &quot;grant_type=client_credentials&quot;,&#10;  encode = &quot;multipart&quot;&#10;);&#10;&#10;&#10;#Extract the access token&#10;token &lt;- paste(&quot;Bearer&quot;, content(req)$access_token)&#10;&#10;&#10;#Request Followers count&#10;url &lt;- paste(&quot;https://api.twitter.com/1.1/users/lookup.json?screen_name=&quot;,target,sep=&quot;&quot;);&#10;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;json &lt;- content(req, as = &quot;text&quot;);&#10;tmp &lt;- fromJSON(json);&#10;num_followers &lt;- tmp[[1]]$followers_count;&#10;&#10;&#10;#If more than 5000 followers split ID request otherwise retrieve all in one request&#10;if (num_followers&lt;5000) {&#10;&#10;&#9;#Request &lt;5000 IDs &#10;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/followers/ids.json?screen_name=&quot;,target,sep=&quot;&quot;);&#10;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;IDs &lt;- fromJSON(json);&#10;&#9;IDlist &lt;- data.frame(IDs$ids);&#10;&#9;&#10;&#9;} else {&#10;&#9;&#10;&#9;#Request batches of IDs&#10;&#9;batches &lt;- ceiling(num_followers/5000);&#10;&#9;batch_size &lt;- ceiling(num_followers/batches);&#10;&#9;curs &lt;- &quot;-1&quot;;&#10;&#9;IDlist &lt;- data.frame();&#10;&#9;for (i in 1:batches) {&#10;&#9;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/followers/ids.json?cursor=&quot;,curs,&quot;&amp;screen_name=&quot;,target,&quot;&amp;count=&quot;,batch_size,sep=&quot;&quot;);&#10;&#9;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;&#9;IDs &lt;- fromJSON(json);&#10;&#9;&#9;tmp &lt;- data.frame(IDs$ids);&#10;&#9;&#9;IDlist &lt;- rbind(IDlist, tmp);&#10;&#9;&#9;curs &lt;- IDs$next_cursor_str;&#10;&#9;&#9;Sys.sleep(60);&#10;&#9;&#9;};&#10;&#9;};&#10;&#10;&#10;#Loop through IDs in batches to retrieve full profiles&#10;batches &lt;- ceiling(num_followers/100);&#10;batch_size &lt;- ceiling(num_followers/batches);&#10;BatchIDcounter &lt;- 0;&#10;FollowersList &lt;- data.frame();&#10;for (i in 1:batches) {&#10;&#9;start &lt;- BatchIDcounter;&#10;&#9;end &lt;- BatchIDcounter+batch_size;&#10;&#9;if (end&gt;length(IDlist[,1])) {&#10;&#9;&#9;end &lt;-length(IDlist[,1]);&#10;&#9;&#9;batch_size &lt;- (end-start);&#10;&#9;};&#10;&#9;BatchIDs &lt;- paste(IDlist[start:end,1],collapse=&quot;,&quot;);&#10;&#9;url &lt;- paste(&quot;https://api.twitter.com/1.1/users/lookup.json?include_entities=false&amp;user_id=&quot;,BatchIDs,sep=&quot;&quot;);&#10;&#9;req &lt;- GET(url, config(httpheader = c(&quot;Authorization&quot; = token)));&#10;&#9;json &lt;- content(req, as = &quot;text&quot;);&#10;&#9;profiles &lt;- fromJSON(json);&#10;&#9;for (j in 1:batch_size) {&#10;&#9;&#9;tmp &lt;- data.table(id_str=&quot;&quot;,name=&quot;&quot;,screen_name=&quot;&quot;,url=&quot;&quot;,profile_image_url=&quot;&quot;,description=&quot;&quot;,location=&quot;&quot;,followers_count=&quot;&quot;,friends_count=&quot;&quot;,statuses_count=&quot;&quot;,created_at=&quot;&quot;);&#10;&#9;&#9;if (length(profiles[[j]]$id_str)==0) {tmp$id_str=&quot;&quot;} else{tmp$id_str=head(profiles[[j]]$id_str[1])};&#10;&#9;&#9;if (length(profiles[[j]]$name)==0) {tmp$name=&quot;&quot;} else{tmp$name=head(profiles[[j]]$name[1])};&#10;&#9;&#9;if (length(profiles[[j]]$screen_name)==0) {tmp$screen_name=&quot;&quot;} else{tmp$screen_name=head(profiles[[j]]$screen_name[1])};&#10;&#9;&#9;if (length(profiles[[j]]$url)==0) {tmp$url=&quot;&quot;} else{tmp$url=head(profiles[[j]]$url[1])};&#10;&#9;&#9;if (length(profiles[[j]]$profile_image_url)==0) {tmp$profile_image_url=&quot;&quot;} else{tmp$profile_image_url=head(profiles[[j]]$profile_image_url[1])};&#10;&#9;&#9;if (length(profiles[[j]]$description)==0) {tmp$description=&quot;&quot;} else{tmp$description=head(profiles[[j]]$description[1])};&#10;&#9;&#9;if (length(profiles[[j]]$location)==0) {tmp$location=&quot;&quot;} else{tmp$location=head(profiles[[j]]$location[1])};&#10;&#9;&#9;if (length(profiles[[j]]$url)==0) {tmp$url=&quot;&quot;} else{tmp$url=head(profiles[[j]]$url[1])};&#10;&#9;&#9;if (length(profiles[[j]]$followers_count)==0) {tmp$followers_count=&quot;&quot;} else{tmp$followers_count=head(profiles[[j]]$followers_count[1])};&#10;&#9;&#9;if (length(profiles[[j]]$friends_count)==0) {tmp$friends_count=&quot;&quot;} else{tmp$friends_count=head(profiles[[j]]$friends_count[1])};&#10;&#9;&#9;if (length(profiles[[j]]$statuses_count)==0) {tmp$statuses_count=&quot;&quot;} else{tmp$statuses_count=head(profiles[[j]]$statuses_count[1])};&#10;&#9;&#9;if (length(profiles[[j]]$created_at)==0) {tmp$created_at=&quot;&quot;} else{tmp$created_at=head(profiles[[j]]$created_at[1])};&#10;&#9;&#9;FollowersList &lt;- rbind(FollowersList, tmp);&#10;&#9;};&#10;&#9;BatchIDcounter &lt;- end;&#10;&#9;Sys.sleep(5);&#10;};&#10;&#10;&#10;&#10;#Output&#10;outdata &lt;- as.data.frame(FollowersList);&#10;"/>
        <enumeration key="inputs">
          <parameter key="name_of_variable" value="indata"/>
        </enumeration>
        <list key="results">
          <parameter key="outdata" value="Data Table"/>
        </list>
      </operator>
      <operator activated="true" breakpoints="before" class="write_excel" compatibility="5.3.015" expanded="true" height="76" name="Write Excel" width="90" x="447" y="30">
        <parameter key="excel_file" value="C:\Documents and Settings\menghii1\Desktop\first.xlsx"/>
        <parameter key="file_format" value="xlsx"/>
      </operator>
      <connect from_op="Generate Data by User Specification" from_port="output" to_op="Execute Script (R)" to_port="input 1"/>
      <connect from_op="Execute Script (R)" from_port="output 1" to_op="Write Excel" to_port="input"/>
      <connect from_op="Write Excel" from_port="through" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Logged
ighyboo
Newbie
*
Posts: 26


« Reply #1 on: October 02, 2014, 12:59:22 PM »

P.S.

Here the log details:

Code:
SEVERE: Process failed: In order to import an R Data Frame as example set the data frame must provide attribute names.
com.rapidminer.operator.UserError: In order to import an R Data Frame as example set the data frame must provide attribute names.
at com.rapidminer.tools.r.translation.ExampleSetTranslator.getAttribute(ExampleSetTranslator.java:174)
at com.rapidminer.tools.r.translation.ExampleSetTranslator.importObject(ExampleSetTranslator.java:105)
at com.rapidminer.tools.r.translation.ExampleSetTranslator.importObject(ExampleSetTranslator.java:64)
at com.rapidminer.operator.r.RScriptOperator.doWork(RScriptOperator.java:217)
at com.rapidminer.operator.Operator.execute(Operator.java:866)
at com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
at com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:711)
at com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:375)
at com.rapidminer.operator.Operator.execute(Operator.java:866)
at com.rapidminer.Process.run(Process.java:949)
at com.rapidminer.Process.run(Process.java:873)
at com.rapidminer.Process.run(Process.java:832)
at com.rapidminer.Process.run(Process.java:827)
at com.rapidminer.Process.run(Process.java:817)
at com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)

Logged
ighyboo
Newbie
*
Posts: 26


« Reply #2 on: February 03, 2015, 02:09:34 PM »

Has anyone looked into this?

I see that the same problem appeared in another post http://rapid-i.com/rapidforum/index.php/topic,7128.msg24720.html#msg24720.
"tennenrishin" reported that this issue only happened when NA values were present however I tested that and in my case it happens even if all the values are present.
I even tried to explicitly declare every column of the DataFrame when creating it but no luck...

This was actually one of the main reasons why I stopped using RM and moved to do most of my work directly in R... But that's a real pity as there are many things that I like about RM.

Any possible solution?
Logged
javemar
Newbie
*
Posts: 23


« Reply #3 on: Today at 04:25:07 AM »

Hello


I had the same problem. What I figured is when you put data from R to Rapidminer you must assure that all character data is a factor. Example


If you have a input variable name as "input" and you want to create a column "newcolumn" with the value  "newvalue"  may you type

input$newcolumn<-"newvalue"

But this expression will give you a error.

However, if you try

input$newcolumn<-as.factor("newvalue" )

Rapidminer will recognize the new attribute










Logged
David A
Global Moderator
Newbie
*****
Posts: 13


« Reply #4 on: Today at 08:13:48 AM »

As a general tip, there is the new R scripting extension, which offers a much smoother integration of R for RapidMiner:
https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_r_scripting
Logged
Pages: [1]
  Print  
 
Jump to: