save custom model in plugin and reuse it

Options
og_dz_disrupt
og_dz_disrupt Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1 Partner

Hi DSS community,

I am trying to create an optimization (not an ML model using sklearn and other libraries) plugin for a specific project. The model I created has .fit() and .predict() functions.

When training the model, I used the .fit() function. How can I use this saved model later with its .predict() function, in the same plugin ?

Small example:

1) Training my model using a first python recipe recipe.py:

my_model=model.fit()

2) Use it with a second recipe using the same plugin in the prediction_recipe.py:

my_prediction=my_model.predict()

Thanks in advance

Tagged:

Answers

  • pmasiphelps
    pmasiphelps Dataiker, Dataiku DSS Core Designer, Registered Posts: 33 Dataiker
    Options

    Hi,

    I'd recommend having two "recipe" components as part of your plugin - one for training, one for scoring.

    Here's an example of how the deep learning image plugin has separate train/score recipes:

    deeplearning-flow.png

    The train recipe has your train dataset as an input, and it outputs a serialized model object (saved as ONNX, h5, pickle, etc.) saved to a managed folder. The score recipe then takes the model folder and your dataset-to-be-scored as inputs, and outputs predictions to a dataset.

    (Note that the deep learning image plugin is now deprecated in favor of deep learning integrated in the regular visual ML interface - but this is still a good example of how you can structure it)

    Here's the github repo for this plugin: https://github.com/dataiku/dss-plugin-deeplearning-image/tree/master/custom-recipes

    In each of the sub-folders here^, there's a separate recipe (for train, score, etc.).

    Best,

    Pat

Setup Info
    Tags
      Help me…