Create an 'training' recipe from YOLOv3 darknet

jax79sg

Hi,

I am assessing Dataiku's flexibility in accommodating very new research specifically in Deep Learning. I am choosing YOLOv3 as my test subject due to its ease of set up and well known outcomes.

Methodology

1. I have created a docker container image based on YOLOv3 darknet. (https://pjreddie.com/darknet/yolo/), to which i can execute YOLOv3 darknet's python codes to train images from a persistent volume.
2. Now i would like to create a recipe/macro in Dataiku that would allow Dataiku users to freely use YOLOv3 without looking at its code.
3. I would imagine the completed recipe to support;
   1. Point the input-dataset to a set of images and labels
   2. Allow users to set different hyperparameters on recipe settings
   3. Retrain YOLOv3 from from the input-dataset and hyperparameter settings
   4. Evaluate the best performing model
   5. Save the model for scoring in other projects

Is this idea possible? If yes, what would be the broad approach to it?

Thank you.

Mattsco

Hi,

You can create a plugin with several recipes to do that. It would appear in the list of recipes of your Dataiku instance and would allow Dataiku users to freely use YOLOv3 without looking at its code, just specifying parameters in the interface.



You can start by our tutorial to do a plugin recipe:

https://academy.dataiku.com/latest/tutorial/plugins/recipe.html



Then, here is an example of plugin to train deep learning models for images using keras/tensorflow:

https://www.dataiku.com/dss/plugins/info/deeplearning-image.html

You'll see there are multiple components in this plugin: recipes, macros and a webapp.

You'll find the code here:

https://github.com/dataiku/dataiku-contrib/tree/master/deeplearning-image-gpu

Matt

jax79sg

Hi,

Not sure if that was what i am looking for, allow me to elaborate.
1. The github core code is mainly written in C++ but the owner has created a simple python wrapper for users to perform training and inference using simple python calls. I have containerized the entire code and installed the required dependencies on the docker container.

2. I do not wish to reproduce the github code on Dataiku. I just want to call the python codes in the docker container. So my python code in the Recipe should be calling the python codes in the docker container, but i don't see how this can be done in the recipe.

Possibly..Maybe install the entire Github code and dependencies as a code env for the recipe?

For comments please.

Mattsco

Yes, in the advanced tab of a python recipe, there is a container configuration that can be setup.
So you could configure a docker container in dataiku administration, with all the code libraries you developed to run YOLO.
Then the code written in the recipe would be executed in this container.