Long Queue Time in Jupter notebook

kai_fang

Recently we start to experience a long queue time when we run multiple block in DataIKU jupyter notebook. The block will show execution queued and it will take a long time until the block get executed and return the result, even when the block only try to calculate simple formulas like 1+1, and the actual exectuion takes 5 ms to finish, while the queue wait time is like 2-20 minutes. This happens only when we execute the code block by block, and not when we do "Restart and Run All" for the whole notebook. As jupyter notebook is designed to explore the data and do some quick analysis and visualization, this slow execution queue really gives a painful experience for development in dataiku.

And not sure why, if I experience a slow execution queue, if i force to end the execution and run the block again, it will immediatedly return me the result, if I make a small modification on the code block, like adding a space.

I am wondering whether this is a common issue for the community or not and if anyone has experience similar issue and know how to fix it, really appreciate for the help.

Thanks.

Operating system used: Linux

Sarina

Hi @kai_fang,

This indeed is unusual. Is it possible that the notebook is still starting up when this happens? If you are executing your notebook on a container, and you are running up against the resource constraints of the cluster, it's possible that it's taking a long time for the pod to be successfully scheduled on your cluster. We can take a look if you open a support ticket with:
- a brief video capture showing the issue/screenshots of the issue
- an instance diagnostic of the DSS, which will need to be created by an admin of your instance.

Thank you!
Sarina

kai_fang

It turns out that DataIku version upgrade ends up including the variable inspector extension as default in the jupyter notebook, and when memory usage increase, variable inspector will ends up causing this latency issue, and the time to run next cell will be a lot slower. So it is mainly a jupyter notebook issue caused by the extension.