Executing notebooks inside a container with UIF activated

DrissiReda
Level 4
Executing notebooks inside a container with UIF activated

Hello,

 

I tried running a jupyter notebook's kernel inside a container while UIF was enabled. And I'm getting this error in the kernel:

Traceback (most recent call last):
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/web.py", line 1592, in _execute
    result = yield result
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/handlers.py", line 81, in post
    user=self.get_current_user(),
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/sessionmanager.py", line 87, in create_session
    DKU_EXTRA_ENV= dku_extra_env)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/sessionmanager.py", line 96, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name, **kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/kernels/kernelmanager.py", line 148, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/manager.py", line 259, in start_kernel
    **kw)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 173, in _launch_kernel
    filename = self.build_description(kernel_cmd, self.impersonate, kernel_cwd, env, kernel_context)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 106, in build_description
    "dss_user" : kernel_context.get('dssUser', os.environ['USER']),
  File "/usr/lib64/python3.6/os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'USER'

 

When I don't activate the UIF I have no problem. So I don't understand why this happens when I enable, despite not trying to run any code locally.

0 Kudos
4 Replies
ATsao
Dataiker

Hi, 

Could you share what code you are trying to run in your notebook? Also, if you run the same in a python recipe, do you get similar errors? If so, please open a support ticket with the corresponding job diagnosis

Thanks,

Andrew

0 Kudos
DrissiReda
Level 4
Author

I'm running the default generated code whenever you create a new notebook. This issue does not occur while running a python recipe. Those work as expected even when UIF is enabled.

0 Kudos
ATsao
Dataiker

The error you are receiving implies that the USER is not properly set on the instance or at least it's not being successfully retrieved from the OS environment variables (os.environ['USER']). If possible,  please try reviewing and printing out the OS environment variables to make sure they are configured appropriately. 

You can potentially try working around the issue by adding the following to the <DATA_DIR>/bin/env-site.sh file:

export USER=<insert_user>

 

0 Kudos
DrissiReda
Level 4
Author

I tried setting that variable manually. But I don't understand why is it needed? Why is the UIF relevant when executing the kernel in a container??

 

And adding the variable manually moves me to this next error:

Traceback (most recent call last):
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/web.py", line 1592, in _execute
    result = yield result
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/handlers.py", line 81, in post
    user=self.get_current_user(),
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/sessionmanager.py", line 87, in create_session
    DKU_EXTRA_ENV= dku_extra_env)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/sessions/sessionmanager.py", line 96, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name, **kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/tornado/gen.py", line 326, in wrapper
    yielded = next(result)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/services/kernels/kernelmanager.py", line 148, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/home/dataiku/dataiku-dss-10.0.5/python36.packages/jupyter_client/manager.py", line 259, in start_kernel
    **kw)
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 201, in _launch_kernel
    check_alive()
  File "/home/dataiku/dataiku-dss-10.0.5/dku-jupyter/packages/notebook/dataiku/kernelmanager.py", line 187, in check_alive
    raise Exception("MUS Wrapper process died !")
Exception: MUS Wrapper process died !

 

I don't want UIF to be relevant at all when I'm using containerized execution. This works correctly for any recipe. But it is broken in notebooks. Is this intentional?

0 Kudos