Community Conundrum 28: News Engagement is live! Read More

SparkR recipe : recipe output from R dataframe

Level 1
SparkR recipe : recipe output from R dataframe

Version: Spark v2.2 

library(SparkR)
library(dataiku)
library(dataiku.spark2)

sc <- sparkR.session()

 

Task:

-Convert SparkR dataframe to R dataframe for various computations 

-Generate Final result in R data-frame

-Generate Recipe output from R data-frame  

 

Syntax used to convert SparkR dataframe to R dataframe:

 df_r  <- collect(df_s)    # this seems to be working 

 

Systax to convert R dataframe to SparkR

result_s <- createDataFrame(sc, df_r)

# Recipe outputs
dkuSparkWriteDataset(result_s, "TestTransform")

 

 

But it seems this syntax does not work, It fails 

 

 

 

 

 

0 Kudos
1 Reply
Dataiker
Dataiker

What is the failure you are getting? Please post the error message you get. 

0 Kudos
A banner prompting to get Dataiku DSS