SparkR recipe : recipe output from R dataframe

vivekkumar

Version: Spark v2.2

library(SparkR)
library(dataiku)
library(dataiku.spark2)

sc <- sparkR.session()

Task:

-Convert SparkR dataframe to R dataframe for various computations

-Generate Final result in R data-frame

-Generate Recipe output from R data-frame

Syntax used to convert SparkR dataframe to R dataframe:

df_r <- collect(df_s) # this seems to be working

Systax to convert R dataframe to SparkR

result_s <- createDataFrame(sc, df_r)

# Recipe outputs
dkuSparkWriteDataset(result_s, "TestTransform")

But it seems this syntax does not work, It fails

Triveni

What is the failure you are getting? Please post the error message you get.