How to add hyperparam optimization support to custom models?

Solved!
dadbuz
Level 3
How to add hyperparam optimization support to custom models?

Hi,

I've built a custom ML model following the documentation [ https://doc.dataiku.com/dss/latest/machine-learning/algorithms/in-memory-python.html#custom-models ] and would like to be able to leverage dataiku builtin cross-validation and hyperparameter search to find optimal parameter settings.

Thanks

 

0 Kudos
1 Solution
Marine
Dataiker

Hi @dadbuz , 

Today, the hyperparameter optimisation does not work for custom python models in auto ML. Grid search will work for plugin models. Converting a custom model into a plugin model is quite straightforward. For more information, you may have a look at this tutorial

Have you tried to package your model as a plugin? If yes, did you encounter any limitations? 

View solution in original post

0 Kudos
6 Replies
Marine
Dataiker

Hi @dadbuz ! 

During the training, you can select K-fold cross test both for default and custom ML models. Go to your Analysis > Design > Train / test set > 

 

You can set up grid search parameters for your custom models by adding fields into the algo.json file. Have a look at this documentation for more information : https://doc.dataiku.com/dss/latest/plugins/reference/prediction-algorithms.html#grid-search 

You can find an example of a custom model implementing managed grid search on Dataiku's github : https://github.com/dataiku/dss-plugin-model-lightgbm/blob/master/python-prediction-algos/model-light... 

 

0 Kudos
dadbuz
Level 3
Author

Hi @Marine,

Well, is there a way to invoke hypeparams optimisation without having to convert my custom model to a plugin?  

0 Kudos
Marine
Dataiker

Hi @dadbuz , 

Today, the hyperparameter optimisation does not work for custom python models in auto ML. Grid search will work for plugin models. Converting a custom model into a plugin model is quite straightforward. For more information, you may have a look at this tutorial

Have you tried to package your model as a plugin? If yes, did you encounter any limitations? 

0 Kudos
camzis
Level 2

Hi 

I have package my custom model as a plugin but I got an issue when parsing parameters. 

You can see the screen of the bug 

Any help would be appreciate ! 

Best 

0 Kudos
Marine
Dataiker

Hi @camzis , 

This looks like a bug that the next DSS version will fix. We are sorry for the inconvenience. Meanwhile, you could drop the grid search and use a DOUBLE parameter as a workaround. 

0 Kudos
cooxky
Level 3

Dear Dataiku Team,

Have there been any updates regarding this topic since 2021?

Or hyperparameter optimisation still does not work for custom python models in auto ML and turning model into a plugin is the only option?

Thank you.