Issue creating Python code-env on a machine with no internet access

rothfuss

The machine I'm working on has no internet connection and I have downloaded and installed all the packages mentioned below on the system by executing this command (for example pyzmq package):

pip install pyzmq-17.1.2-cp27-cp27mu-manylinux1_x86_64.whl -f ./ --no-index



However, code-env is unable to read this and throws this error:



*********************************************************
*********************************************************
*
* create empty env
*
* command : /var/opt/dss/bin/python /opt/dss/dataiku-dss-5.1.0/scripts/virtualenv/virtualenv.py -p python2.7 /var/opt/dss/code-envs/python/test2
*
*********************************************************
*********************************************************
New python executable in /var/opt/dss/code-envs/python/test2/bin/python2.7
Also creating executable in /var/opt/dss/code-envs/python/test2/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /bin/python2.7
*********************************************************
*********************************************************
*
* list packages
*
* command : /var/opt/dss/code-envs/python/test2/bin/python /var/opt/dss/code-envs/python/test2/bin/pip freeze -l
*
*********************************************************
*********************************************************
Installing from Pip requirements:
pandas==0.23.4
python-dateutil==2.7.4
pytz==2018.6
requests==2.20.1
decorator==4.3.0
ipykernel==4.8.2
ipython_genutils==0.2.0
jupyter_client==5.2.3
jupyter_core==4.4.0
pexpect==4.6.0
pickleshare==0.7.5
ptyprocess==0.6.0
pyzmq==17.1.2
simplegeneric==0.8.1
tornado==5.1.1
traitlets==4.3.2
*********************************************************
*********************************************************
*
* install packages
*
* command : /var/opt/dss/code-envs/python/test2/bin/python /var/opt/dss/code-envs/python/test2/bin/pip install --upgrade -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt
*
*********************************************************
*********************************************************
Collecting pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2))
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
Could not find a version that satisfies the requirement pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2)) (from versions: )
No matching distribution found for pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2))

Clément_Stenac

Hi,

Code envs will use "pip install -r requirements.txt" with a requirements.txt made of the packages you requested.

By design, code envs will never try to reuse packages that are already installed at the system level (what you did with your wheel install), since the goal of code envs in DSS is to provide isolated environments.

You can add custom options to the pip command line by going into the code env settings, unchecking "use global settings", and adding command line options. From the doc of pip, you should be able to install from your wheel by adding the following options:

--no-index
--find-links=/path/to/your/wheels

(see https://pip.pypa.io/en/stable/user_guide/#id14)

rothfuss

Got it, I had to download and place each dependency, and it worked. But it throws me another error post the pip phase:

Collecting ipykernel==4.8.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting ipython_genutils==0.2.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 9))
Collecting jupyter_client==5.2.3 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 10))
Collecting jupyter_core==4.4.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 11))
Collecting pexpect==4.6.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 12))
Collecting pickleshare==0.7.5 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting ptyprocess==0.6.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 14))
Collecting pyzmq==17.1.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 15))
Collecting simplegeneric==0.8.1 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 16))
Collecting tornado==5.1.1 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting traitlets==4.3.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 18))
Requirement already satisfied, skipping upgrade: numpy>=1.9.0 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from pandas==0.23.4->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 2)) (1.14.2)
Requirement already satisfied, skipping upgrade: six>=1.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from python-dateutil==2.7.4->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 3)) (1.11.0)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (2.7)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (1.24.1)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (2018.11.29)
Collecting ipython>=4.0.0 (from ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting pathlib2; python_version in "2.6 2.7 3.2 3.3" (from pickleshare==0.7.5->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting futures (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting singledispatch (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting backports-abc>=0.4 (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting enum34; python_version == "2.7" (from traitlets==4.3.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 18))
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting pygments (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Requirement already satisfied, skipping upgrade: setuptools>=18.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8)) (40.6.3)
Collecting prompt-toolkit<2.0.0,>=1.0.4 (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting scandir; python_version < "3.5" (from pathlib2; python_version in "2.6 2.7 3.2 3.3"->pickleshare==0.7.5->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Building wheels for collected packages: scandir
Running setup.py bdist_wheel for scandir: started
Running setup.py bdist_wheel for scandir: finished with status 'error'
Complete output from command /var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-u4MHKm --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying scandir.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_scandir' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
_scandir.c:14:20: fatal error: Python.h: No such file or directory
#include
^
compilation terminated.
error: command 'gcc' failed with exit status 1

----------------------------------------
Running setup.py clean for scandir
Failed to build scandir
Installing collected packages: scandir, pathlib2, backports.shutil-get-terminal-size, simplegeneric, pygments, ptyprocess, pexpect, enum34, ipython-genutils, traitlets, wcwidth, prompt-toolkit, pickleshare, ipython, futures, singledispatch, backports-abc, tornado, pyzmq, jupyter-core, jupyter-client, ipykernel
Running setup.py install for scandir: started
Running setup.py install for scandir: finished with status 'error'
Complete output from command /var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ry4fxI/install-record.txt --single-version-externally-managed --compile --install-headers /var/opt/dss/code-envs/python/test2/include/site/python2.7/scandir:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
copying scandir.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_scandir' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
_scandir.c:14:20: fatal error: Python.h: No such file or directory
#include
^
compilation terminated.
error: command 'gcc' failed with exit status 1

----------------------------------------
Failed building wheel for scandir
notebook 5.4.0-dku-5.1-1 requires jinja2, which is not installed.
notebook 5.4.0-dku-5.1-1 requires nbconvert, which is not installed.
notebook 5.4.0-dku-5.1-1 requires nbformat, which is not installed.
notebook 5.4.0-dku-5.1-1 requires Send2Trash, which is not installed.
notebook 5.4.0-dku-5.1-1 requires terminado>=0.8.1, which is not installed.
Command "/var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ry4fxI/install-record.txt --single-version-externally-managed --compile --install-headers /var/opt/dss/code-envs/python/test2/include/site/python2.7/scandir" failed with error code 1 in /tmp/pip-install-jSsFWK/scandir/

Can you please shed some light on this?

rothfuss

The issue was with my python installation for this - I was missing the python-devel package. installed that, and this is working fine now.

Thanks for the help, Clément