How can I use spark MLLib in a custom model ?
Options
HugoDupeux
Registered Posts: 3 ✭✭✭✭
Good morning,
I want to create a custom model using MLLib (because I need to use Frequent Pattern Mining algorithms who are already in there).
I found some documentation on how create a custom model :
- https://doc.dataiku.com/dss/latest/machine-learning/algorithms/in-memory-python.html
- https://academy.dataiku.com/custom-models-in-visual-ml/50678
- https://doc.dataiku.com/dss/latest/machine-learning/algorithms/mllib.html
But none of them explain how to do it (in my comprehension). Python and sklearn are well explained and shown but not MLLib. I wonder if there is a code example I could use to help me or a more sepcific documentation.
I am using dataiku V7, not 8. I don't know if there is an impact on custom model.
Thank you.
Kind Regards,
Hugo
Answers
-
Hello,
Inside of the custom model editor there is a sample provided, I have pasted it below.
// This sample code uses a standard MLlib algorithm, the RandomForestRegressor. // import the Estimator from spark.ml import org.apache.spark.ml.regression.RandomForestRegressor // instantiate the Estimator new RandomForestRegressor() .setLabelCol("ot_avg") // Must be the target column .setFeaturesCol("__dku_features") // Must always be __dku_features .setPredictionCol("prediction") // Must always be prediction .setNumTrees(50) .setMaxDepth(8)
