Running code in a notebook unsupervised

ben_p

Hi everyone,

I have a long running python Job that I would like to run in a notebook on DSS, with the idea being that if I shutdown by PC the job will continue to run (by DSS is installed on a cloud VM).

I tested this out but then I closed the window the job stopped too, is this expected behaviour and is there any way to achieve background running of my code in a notebook on DSS?

Thanks,
Ben

tgb417

@ben_p

l don’t know if this shutdown was expected behavior. I have not had this problem.

However, in reading your post I was wondering why you were not converting your notebook to a recipe. I believe as a recipe, a set of notebook code should run to completion. At least that has been my experience.

As you may be aware you can very quickly convert notebooks too an from recipes with a button at the top if the recipe and notebook.

As I write this note I’m wondering about your use case? Are you doing something like trying to train a neural network model that may take quite some time and you are still very actively in development? Can you share a bit more?

ben_p

Hi @tgb417
- I've actually gone down the route of making it a recipe to get it more stable. In this case it is a "throw-away" bit of code that I only need to use once - I am sending a bunch of API requests to a HTTP endpoint and it takes several hours to complete the whole job.

In may indeed by the case that I should just be using a recipe to do this, rather than a notebook

Ben

tgb417

I'd be interested in a perspective from any of the Dataikuers who might be online. Do you have any insight into this about the Notebook shutdown? And a recommendation for these one-off processes.

Marlan

I commonly kick off notebooks to run overnight and then log out of DSS. They run just fine but the cell outputs aren't captured when you log back in and bring up the notebook. The notebook has run and you can re-run cells that produce output. Not ideal by any means but usually workable. I've also used notebook logging to capture output which can be helpful but isn't perfect either. I don't have a way to stay logged in through the night for various reasons so have struggled with this (i.e., primarily missing the cell output) for some time.

Marlan