Plugin Development in Jupyter

gblack686
Level 4
Plugin Development in Jupyter

I'm trying to test Python  code in Jupyter while developing my plugin. How can I access plugin objects through notebooks?

Struggling to do things like access the plugin inputs.  Is this possible?

 

1 Reply
Alex_Combessie
Dataiker Alumni

[EDITED FOR COMPLETENESS]

Hi,

When developing a plugin, I recommend 2 types of work methodologies:

1. Develop the backend of the plugin in a Jupyter notebook without the plugin-specific interface.

That way, you will be able to test input/output as regular python variables. Easy to prototype ๐Ÿ™‚

Once you are satisfied with the notebook, try and factor your code as functions and/or classes. This is very important, as this will allow separating the "application logic" from the "interface logic". It will make your whole development experience a lot easier.

If you do that, then turning your notebook into a plugin is very simple: copy the functions/classes in the plugin "python-lib" folder. Your plugin component code (for instance recipe.py) will just contain the "input/output glue" which is plugin-specific.

2. Test and enhance your plugin prototype

Here there are several methods which can choose according to your preferences. However, one thing to keep in mind is that you won't be able to directly access the plugin inputs from the interface. The reason is that by design, plugin components are executed as distinct processes.

A first option is to continue using your original Jupyter notebook, calling the plugin libraries you have created using these methodshttps://doc.dataiku.com/dss/latest/plugins/reference/other.html?highlight=lib#shared-code.

What I often do as a plugin developer is to maintain one Python notebook per plugin component, replicating the input/output logic from the plugin. This works well for simple plugins.

For a more complete development experience, I highly recommend our IDE integrations:

- VSCode: https://academy.dataiku.com/latest/tutorial/code/vscode.html

- PyCharm: https://academy.dataiku.com/latest/tutorial/code/pycharm.html

- Sublime Text: https://academy.dataiku.com/latest/tutorial/code/sublime.html

You will be able to directly modify the plugin code (libraries and components) and test it in DSS by executing the plugin components. I often have two windows: one with the IDE and one with a DSS project to test my plugin, side-by-side. You can use logging statements to monitor what's happening. 

The last step is Quality Assurance: test your whole plugin again on a fresh new DSS project, possibly on another DSS instance. Try and find as many bugs and edge cases as possible, and fix 'em!

Hope it helps,

Alex