Executing notebooks inside a container with UIF activated
Hello,
I tried running a jupyter notebook's kernel inside a container while UIF was enabled. And I'm getting this error in the kernel:
Traceback (most recent call last): File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/web.py", line 1592, in _execute result = yield result File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/handlers.py", line 81, in post user=self.get_current_user(), File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/sessionmanager.py", line 87, in create_session DKU_EXTRA_ENV= dku_extra_env) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/sessionmanager.py", line 96, in start_kernel_for_session self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name, **kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/kernels/kernelmanager.py", line 148, in start_kernel super(MappingKernelManager, self).start_kernel(**kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel km.start_kernel(**kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/manager.py", line 259, in start_kernel **kw) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 173, in _launch_kernel filename = self.build_description(kernel_cmd, self.impersonate, kernel_cwd, env, kernel_context) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 106, in build_description "dss_user" : kernel_context.get('dssUser', os.environ['USER']), File "/usr/lib64/python3.6/os.py", line 669, in __getitem__ raise KeyError(key) from None KeyError: 'USER'
When I don't activate the UIF I have no problem. So I don't understand why this happens when I enable, despite not trying to run any code locally.
Answers
-
Hi,
Could you share what code you are trying to run in your notebook? Also, if you run the same in a python recipe, do you get similar errors? If so, please open a support ticket with the corresponding job diagnosis.
Thanks,
Andrew
-
I'm running the default generated code whenever you create a new notebook. This issue does not occur while running a python recipe. Those work as expected even when UIF is enabled.
-
The error you are receiving implies that the USER is not properly set on the instance or at least it's not being successfully retrieved from the OS environment variables (os.environ['USER']). If possible, please try reviewing and printing out the OS environment variables to make sure they are configured appropriately.
You can potentially try working around the issue by adding the following to the <DATA_DIR>/bin/env-site.sh file:
export USER=<insert_user>
-
I tried setting that variable manually. But I don't understand why is it needed? Why is the UIF relevant when executing the kernel in a container??
And adding the variable manually moves me to this next error:
Traceback (most recent call last): File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/web.py", line 1592, in _execute result = yield result File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/handlers.py", line 81, in post user=self.get_current_user(), File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/sessionmanager.py", line 87, in create_session DKU_EXTRA_ENV= dku_extra_env) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/sessionmanager.py", line 96, in start_kernel_for_session self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name, **kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 326, in wrapper yielded = next(result) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/kernels/kernelmanager.py", line 148, in start_kernel super(MappingKernelManager, self).start_kernel(**kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel km.start_kernel(**kwargs) File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/manager.py", line 259, in start_kernel **kw) File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 201, in _launch_kernel check_alive() File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 187, in check_alive raise Exception("MUS Wrapper process died !") Exception: MUS Wrapper process died !
I don't want UIF to be relevant at all when I'm using containerized execution. This works correctly for any recipe. But it is broken in notebooks. Is this intentional?