How to add hyperparam optimization support to custom models?

[dadbuz]

Hi,

I've built a custom ML model following the documentation [ https://doc.dataiku.com/dss/latest/machine-learning/algorithms/in-memory-python.html#custom-models ] and would like to be able to leverage dataiku builtin cross-validation and hyperparameter search to find optimal parameter settings.

Thanks

[Marine]

Hi @dadbuz ! 

During the training, you can select K-fold cross test both for default and custom ML models. Go to your Analysis > Design > Train / test set > K-fold cross-test

You can set up grid search parameters for your custom models by adding fields into the algo.json file. Have a look at this documentation for more information : https://doc.dataiku.com/dss/latest/plugins/reference/prediction-algorithms.html#grid-search 

You can find an example of a custom model implementing managed grid search on Dataiku's github : https://github.com/dataiku/dss-plugin-model-lightgbm/blob/master/python-prediction-algos/model-light... 

[dadbuz]

Hi @Marine,

Well, is there a way to invoke hypeparams optimisation without having to convert my custom model to a plugin?  

[Marine · Accepted solution]

Hi @dadbuz , 

Today, the hyperparameter optimisation does not work for custom python models in auto ML. Grid search will work for plugin models. Converting a custom model into a plugin model is quite straightforward. For more information, you may have a look at this tutorial. 

Have you tried to package your model as a plugin? If yes, did you encounter any limitations? 

[camzis]

Hi 

I have package my custom model as a plugin but I got an issue when parsing parameters. 

You can see the screen of the bug 

Any help would be appreciate ! 

Best 

[Marine]

Hi @camzis , 

This looks like a bug that the next DSS version will fix. We are sorry for the inconvenience. Meanwhile, you could drop the grid search and use a DOUBLE parameter as a workaround.