Object detection code env: Could not build container image for environment
Hi dataiku community!
I am trying to use Dataiku's Object Detection example project to train my own model: Project description - Wiki | Dataiku
However, I am having issues when attempting to create the suggested code env: py_38_sample_object_detection (Python) - Code Envs | Dataiku for container configurations
I have created an code env exactly equivalent to the above linked, but when I choose:
Containerized execution --> Build for selected container configurations
I keep getting the same error:
""Could not build container image for environment
docker failed (exit code: 1)""
torch==1.8.1torchvision==0.9.1tornado==5.1.1tqdm==4.65.0traitlets==4.3.3typed-ast==1.5.4typing_extensions==4.6.3urllib3==1.26.6wcwidth==0.2.6Werkzeug==2.2.3yacs==0.1.8zipp==3.15.0Installed kernelspec py-dku-venv-object-detection-new in /data/dataiku/dss_data/jupyter-run/jupyter/kernels/py-dku-venv-object-detection-newInstalled kernelspec py-dku-containerized-venv-object-detection-new-compute-design-gpu in /data/dataiku/dss_data/jupyter-run/jupyter/kernels/py-dku-containerized-venv-object-detection-new-compute-design-gpuAdding resources environment variables to Jupyter kernel.json ...Building docker image$ docker build -t dataiku-container-exec-base-cuda-cicd-11.2.0-pyenv-object-detection-new:r-2023-06-23-13-28-28-264 --force-rm /data/dataiku/dss_data/tmp/docker-image/codeenv-object-detection-new6461684965385574816 -t crk8sdataikuprod.azurecr.io/dataiku-container-exec-base-cuda-cicd-11.2.0-pyenv-object-detection-new:r-2023-06-23-13-28-28-264Sending build context to Docker daemon 4.096kBStep 1/13 : FROM dataiku-container-exec-base-cuda-cicd:11.2.0 ---> 382ce570ed33Step 2/13 : USER root ---> Using cache ---> b9b2e9f51916Step 3/13 : WORKDIR /opt/dataiku ---> Using cache ---> 5627184cd74cStep 4/13 : ENV PYTHONPATH= ---> Using cache ---> 5a1effc1f7acStep 5/13 : ENV R_LIBS_USER= ---> Using cache ---> 51131f18b387Step 6/13 : ENV DKU_IMAGE_BUILD_TIMESTAMP=1687526908264 ---> Running in 3f6026947273Removing intermediate container 3f6026947273 ---> 07f6b26d6c01Step 7/13 : RUN ["bin/python", "build/virtualenv.pyz", "-p", "python3.7", "code-env"] ---> Running in 168348aa0cc5created virtual environment CPython3.7.15.final.0-64 in 650ms creator CPython3Posix(dest=/opt/dataiku/code-env, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv) added seed packages: pip==22.1.2, setuptools==62.6.0, wheel==0.37.1 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivatorRemoving intermediate container 168348aa0cc5 ---> 50dd2916aff0Step 8/13 : COPY code-env/pip.packages.txt code-env/ ---> c5b877cf51e1Step 9/13 : RUN ["/opt/dataiku/code-env/bin/python", "-m", "pip", "install", "--extra-index-url", "https://amrab@orsted.com:cmVmdGtuOjAxOjE3MTYwMjAyMDg6ZGRNMEVpVkdyNUM5QzRZY3NuQ2JKUkcxTlNj@orsted.jfrog.io/artifactory/incubator-pytorch-remote", "-r", "code-env/pip.packages.txt"] ---> Running in 4eaf662bfd5fLooking in indexes: https://pypi.org/simple, https://amrab@orsted.com:****@orsted.jfrog.io/artifactory/incubator-pytorch-remoteCollecting git+https://github.com/facebookresearch/detectron2.git (from -r code-env/pip.packages.txt (line 8)) Cloning https://github.com/facebookresearch/detectron2.git to /tmp/pip-req-build-cmdt0cyg[91m Running command git clone --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-cmdt0cyg[0m Resolved https://github.com/facebookresearch/detectron2.git to commit eddac5d5b4fce9b5eba5518f9ee348fa3a531407 Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error'[91m error: subprocess-exited-with-error à python setup.py egg_info did not run successfully. â exit code: 1 â°â> [6 lines of output] Traceback (most recent call last): File "<string>", line 36, in <module> File "<pip-setuptools-caller>", line 34, in <module> File "/tmp/pip-req-build-cmdt0cyg/setup.py", line 10, in <module> import torch ModuleNotFoundError: No module named 'torch' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip.[0m[91merror: metadata-generation-failedà Encountered error while generating package metadata.â°â> See above for output.note: This is an issue with the package mentioned above, not pip.hint: See above for details.[0m[notice] A new release of pip available: 22.1.2 -> 23.1.2[notice] To update, run: /opt/dataiku/code-env/bin/python -m pip install --upgrade pipRemoving intermediate container 4eaf662bfd5fThe command '/opt/dataiku/code-env/bin/python -m pip install --extra-index-url https://amrab@orsted.com:cmVmdGtuOjAxOjE3MTYwMjAyMDg6ZGRNMEVpVkdyNUM5QzRZY3NuQ2JKUkcxTlNj@orsted.jfrog.io/artifactory/incubator-pytorch-remote -r code-env/pip.packages.txt' returned a non-zero code: 1
Does anybody know how to solve this? it works without the container figurations...
Operating system used: Windows 10
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,215 Dataiker
Hi Simone,
From the lines, you shared :
line 10, in <module> import torch ModuleNotFoundError: No module named 'torch'It's failing when trying to install git+https://github.com/facebookresearch/detectron2.git
Changing this to https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.8/detectron2-0.5%2Bcu101-cp38-cp38-linux_x86_64.whl instead seems to work finetorch==1.8.1 torchvision==0.9.1 pyyaml==5.1 opencv-python==4.6.0.66 dash==2.6.1 dash-bootstrap-components==1.2.1 https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.8/detectron2-0.5+cu101-cp38-cp38-linux_x86_64.whl
It appears this was broken upstream based on this threads so when this originally created it would have worked but no longer work : https://github.com/facebookresearch/detectron2/issues/4386