building a custom docker image

Options
mogul
mogul Registered Posts: 2 ✭✭✭

I want to run dataiku in docker, and the host has a few GPU's which I want to use. So, it seems I have to build my own image for docker.

This is my recipe: (to keep it simple, and make the build faster, I have omitted CUDA and R for now)

mkdir dataiku-buildcd dataiku-buildwget https://cdn.downloads.dataiku.com/public/dss/8.0.2/dataiku-dss-8.0.2.tar.gztar xzf dataiku-dss-8.0.2.tar.gzsudo dataiku-dss-8.0.2/scripts/install/install-deps.shdataiku-dss-8.0.2/installer.sh -d DATA_DIR -p 11000DATA_DIR/bin/dssadmin build-base-image --type container-exec --without-cuda --without-r

Let it cook for at few minutes, and then lets check:

$ docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEdku-exec-base-sb59tjz4ll5axakfj8fhfv6g dss-8.0.2 91329c9f35b1 11 seconds ago 1.39GBcentos 7 8652b9f0cb4c 7 days ago 204MBdocker_web latest fdfe96bedceb 11 months ago 6.27GBjupyter/datascience-notebook 17aba6048f44 1bb52a350a5a 23 months ago 6.27GB

ah yes, good, there on top, 11 seconds old.

Now, lets fire it up to see whats happens:

$ docker run -p 10000:10000 dku-exec-base-sb59tjz4ll5axakfj8fhfv6g:dss-8.0.2[2020-11-21 15:06:25,794] [1/MainThread] [INFO] [root] Fetching job definitionTraceback (most recent call last):File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_mainInstalling debugging signal handler"__main__", fname, loader, pkg_name)File "/usr/lib64/python2.7/runpy.py", line 72, in _run_codeexec code in run_globalsFile "/opt/dataiku/python/dataiku/container/runner.py", line 483, in <module>execution_id = sys.argv[1]IndexError: list index out of range

That obviously did not work.

Anyone else having success building and running own dataiku docker images?

Answers

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 366 Dataiker
    Options

    Hi @mogul

    Not sure I understand what you are trying to achieve with those steps.

    If you are looking to utilize GPUs in your ML tasks, you can just start DSS (as you have already installed it), create code environment and choose GPU during model training:

    https://doc.dataiku.com/dss/latest/machine-learning/deep-learning/runtime-gpu.html

    If you are interested in containerized execution with CUDA support, you will need to set up and configure the container engine (either Kubernetes or Docker) in DSS:

    https://doc.dataiku.com/dss/latest/containers/concepts.html

  • mogul
    mogul Registered Posts: 2 ✭✭✭
    Options

    The latter. I want to run containerised (docker) with CUDA support.

    To do so I followed the guide over here: https://doc.dataiku.com/dss/latest/containers/custom-base-images.html

    Which results in a docker image that does not work.

    Inspecting with nvidia-smi on the docker host reveals two GPU's

    $ nvidia-smiMon Nov 23 11:28:25 2020+-----------------------------------------------------------------------------+| NVIDIA-SMI 455.28 Driver Version: 455.28 CUDA Version: 11.1 ||-------------------------------+----------------------+----------------------+| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. || | | MIG M. ||===============================+======================+======================|| 0 TITAN RTX On | 00000000:01:00.0 Off | N/A || 41% 36C P8 10W / 280W | 16MiB / 24217MiB | 0% Default || | | N/A |+-------------------------------+----------------------+----------------------+| 1 TITAN RTX Off | 00000000:21:00.0 Off | N/A || 41% 34C P8 13W / 280W | 5MiB / 24220MiB | 0% Default || | | N/A |+-------------------------------+----------------------+----------------------++-----------------------------------------------------------------------------+| Processes: || GPU GI CI PID Type Process name GPU Memory || ID ID Usage ||=============================================================================|| 0 N/A N/A 1371 G /usr/lib/xorg/Xorg 9MiB || 0 N/A N/A 1595 G /usr/bin/gnome-shell 4MiB || 1 N/A N/A 1371 G /usr/lib/xorg/Xorg 4MiB || 1 N/A N/A 1595 G /usr/bin/gnome-shell 0MiB |+-----------------------------------------------------------------------------+

    doing the same inside the container running out dataiku instance:

    $ docker exec -it dataiku_dataiku_1 /bin/bash[dataiku@5ca65e340f6a ~]$ nvidia-smibash: nvidia-smi: command not found

    Which was why I thought I had to build a new docker image.

    Ah yes, the non CUDA image we are currently running is the one I found on dockerhub: https://hub.docker.com/r/dataiku/dss/

Setup Info
    Tags
      Help me…