Long Queue Time in Jupter notebook

Options
kai_fang
kai_fang Registered Posts: 6 ✭✭

Recently we start to experience a long queue time when we run multiple block in DataIKU jupyter notebook. The block will show execution queued and it will take a long time until the block get executed and return the result, even when the block only try to calculate simple formulas like 1+1, and the actual exectuion takes 5 ms to finish, while the queue wait time is like 2-20 minutes. This happens only when we execute the code block by block, and not when we do "Restart and Run All" for the whole notebook. As jupyter notebook is designed to explore the data and do some quick analysis and visualization, this slow execution queue really gives a painful experience for development in dataiku.

And not sure why, if I experience a slow execution queue, if i force to end the execution and run the block again, it will immediatedly return me the result, if I make a small modification on the code block, like adding a space.

I am wondering whether this is a common issue for the community or not and if anyone has experience similar issue and know how to fix it, really appreciate for the help.

Thanks.

Operating system used: Linux

Answers

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    edited July 5
    Options

    Hi @kai_fang,

    This indeed is unusual. Is it possible that the notebook is still starting up when this happens? If you are executing your notebook on a container, and you are running up against the resource constraints of the cluster, it's possible that it's taking a long time for the pod to be successfully scheduled on your cluster. We can take a look if you open a support ticket with:
    - a brief video capture showing the issue/screenshots of the issue
    - an instance diagnostic of the DSS, which will need to be created by an admin of your instance.

    Thank you!
    Sarina

  • kai_fang
    kai_fang Registered Posts: 6 ✭✭
    Options

    It turns out that DataIku version upgrade ends up including the variable inspector extension as default in the jupyter notebook, and when memory usage increase, variable inspector will ends up causing this latency issue, and the time to run next cell will be a lot slower. So it is mainly a jupyter notebook issue caused by the extension.

Setup Info
    Tags
      Help me…