Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Version: Spark v2.2
library(SparkR)
library(dataiku)
library(dataiku.spark2)
sc <- sparkR.session()
Task:
-Convert SparkR dataframe to R dataframe for various computations
-Generate Final result in R data-frame
-Generate Recipe output from R data-frame
Syntax used to convert SparkR dataframe to R dataframe:
df_r <- collect(df_s) # this seems to be working
Systax to convert R dataframe to SparkR
result_s <- createDataFrame(sc, df_r)
# Recipe outputs
dkuSparkWriteDataset(result_s, "TestTransform")
But it seems this syntax does not work, It fails
What is the failure you are getting? Please post the error message you get.