jupyter-run data directory

manuelberbig

Hello,

some of my jupyter-run directories of single notebooks have a huge size. What exaclty is stored in these files. Is it an active session of a jupyter notebook? Some of these directories are very large for notebooks I didn't use for months. I also unloaded the session for these notebooks.

I just found the description in https://doc.dataiku.com/dss/latest/operations/datadir.html. Are these directories deleted internal?

I am looking forward for your answer!

Greetings

JordanB

Hi @manuelberbig
,

The jupyter-run directory requires manual clean up by the DSS user as it is not done automatically. Just note that by cleaning this directory, users will lose the data stored in CWD (current working directory). This storage accrues when users save files in the notebooks without specifying a path - for example, when a user writes from a notebook without specifying a path, the file is stored in jupyter-run/dku-workdirs. You can delete files located in jupyter-run/dku-workdirs/, however, it would "reset" any work done by your users in their respective notebooks if they wrote files locally. IE: wget commands, saved models, etc.

In addition to cleaning the jupyter-run directory, we recommend reviewing the following documentation on safely removing nonessential files, such as old job logs and tmp files, to clear up disk space. This includes automating cleanup tasks through the use of DSS macros:
- https://doc.dataiku.com/dss/latest/operations/disk-usage.html
- https://doc.dataiku.com/dss/latest/operations/disk-usage.html#automating-cleanup-tasks-through-dss-macros

Please let us know if you have any further questions.

Thanks!

Jordan

Tanguy

Hi @JordanB, how can non-admin DSS users clean up this folder ?

Turribeach

https://community.dataiku.com/discussion/comment/45475#Comment_45475

They can't. Only admins can. In v13.3.0 there is a new API to clear Jupyter outputs, likely the most common cause of huge notebooks. Not sure if users can execute it, do the test and let us know…

https://developer.dataiku.com/latest/api-reference/python/other-administration.html#dataikuapi.dss.jupyternotebook.DSSJupyterNotebook.clear_outputs

yonghyun

https://community.dataiku.com/discussion/comment/45475#Comment_45475

사용자도 @JordanB 가 언급한 macros를 사용해서 지울 수 있는 것으로 알고 있습니다.

대신 admin은 dss전체에 대해 삭제 권한이 있지만

사용자는 자기 자신이 만든 자원(log, cache등)에 대해서 만 삭제 할 수 있을 것 입니다.

저도 완벽하게 해당 내용을 파악 하고 있지는 않으니 시도해보고 테스트 해보셔야 합니다.