The Dataiku Frontrunner Awards have just launched to recognize your achievements! Submit Your Entry

How can I use spark MLLib in a custom model ?

HugoDupeux
Level 1
How can I use spark MLLib in a custom model ?

Good morning,

I want to create a custom model using MLLib (because I need to use Frequent Pattern Mining algorithms who are already in there).

I found some documentation on how create a custom model : 

But none of them explain how to do it (in my comprehension). Python and sklearn are well explained and shown but not MLLib. I wonder if there is a code example I could use to help me or a more sepcific documentation. 

I am using dataiku V7, not 8. I don't know if there is an impact on custom model. 

Thank you.

Kind Regards,

Hugo

 

 

0 Kudos
1 Reply
Triveni
Dataiker
Dataiker

Hello, 

 

Inside of the custom model editor there is a sample provided, I have pasted it below.

 

// This sample code uses a standard MLlib algorithm, the RandomForestRegressor.

// import the Estimator from spark.ml
import org.apache.spark.ml.regression.RandomForestRegressor

// instantiate the Estimator
new RandomForestRegressor()
   .setLabelCol("ot_avg")  // Must be the target column
   .setFeaturesCol("__dku_features")  // Must always be __dku_features
   .setPredictionCol("prediction")  // Must always be prediction
   .setNumTrees(50)
   .setMaxDepth(8)
0 Kudos
A banner prompting to get Dataiku DSS