How to add hyperparam optimization support to custom models?
Hi,
I've built a custom ML model following the documentation [ https://doc.dataiku.com/dss/latest/machine-learning/algorithms/in-memory-python.html#custom-models ] and would like to be able to leverage dataiku builtin cross-validation and hyperparameter search to find optimal parameter settings.
Thanks
Best Answer
-
Hi @dadbuz
,Today, the hyperparameter optimisation does not work for custom python models in auto ML. Grid search will work for plugin models. Converting a custom model into a plugin model is quite straightforward. For more information, you may have a look at this tutorial.
Have you tried to package your model as a plugin? If yes, did you encounter any limitations?
Answers
-
Hi @dadbuz
!During the training, you can select K-fold cross test both for default and custom ML models. Go to your Analysis > Design > Train / test set >
You can set up grid search parameters for your custom models by adding fields into the algo.json file. Have a look at this documentation for more information : https://doc.dataiku.com/dss/latest/plugins/reference/prediction-algorithms.html#grid-search
You can find an example of a custom model implementing managed grid search on Dataiku's github : https://github.com/dataiku/dss-plugin-model-lightgbm/blob/master/python-prediction-algos/model-lightgbm_classification/algo.json
-
Hi
I have package my custom model as a plugin but I got an issue when parsing parameters.
You can see the screen of the bug
Any help would be appreciate !
Best
-
Hi @camzis
,This looks like a bug that the next DSS version will fix. We are sorry for the inconvenience. Meanwhile, you could drop the grid search and use a DOUBLE parameter as a workaround.
-
Dear Dataiku Team,
Have there been any updates regarding this topic since 2021?
Or hyperparameter optimisation still does not work for custom python models in auto ML and turning model into a plugin is the only option?
Thank you.
