Can't create a code env from a newer Python version (local DSS installation)

Edvin

Hi, I installed the local free version of Dataiku on macOS 15.0.1. The Dataiku DSS version is 13.2.1.

By default, I can only create Python code env for Python 3.7. The rest of the Python versions are marked as "Not available" in the code env creation form.

I need a higher version than Python 3.7 to create a code env for Dataiku's Question Answering with GPT-3 sample project.

I tried to create a new code env with the following "Custom (lookup in PATH)" values (the deployment type is "Managed by DSS"):

/opt/homebrew/bin/python3.11 throws this error:

Warnings were encountered

    WARNING
    ERR_CODEENV_UPDATE_FAILED
    Environment update failed
    Initial code env packages installation failed
    WARNING
    ERR_CODEENV_JUPYTER_SUPPORT_INSTALL_FAILED
    Jupyter support installation failed
    Cannot install jupyter support

created virtual environment CPython3.11.10.final.0-64 in 586ms  creator CPython3macOsBrew(dest=/Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test4, clear=False, no_vcs_ignore=False, global=False)  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/edvin.mocibob/Library/Application Support/virtualenv)    added seed packages: pip==24.2, setuptools==74.1.2, wheel==0.44.0  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivatorUpdating code environment according to spec ...Installing from Pip requirementsIgnoring ipykernel: markers 'python_version < "3"' don't match your environmentCollecting pandas<2.3,>=2.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 2))  Using cached pandas-2.2.3-cp311-cp311-macosx_11_0_arm64.whl.metadata (89 kB)Collecting numpy<1.27 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 3))  Using cached numpy-1.26.4-cp311-cp311-macosx_11_0_arm64.whl.metadata (114 kB)Collecting python-dateutil<3,>=2.8.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 4))  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)Collecting urllib3<2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 5))  Using cached urllib3-1.26.20-py2.py3-none-any.whl.metadata (50 kB)Collecting requests<3 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 6))  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)Collecting decorator==4.4.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 8))  Using cached decorator-4.4.2-py2.py3-none-any.whl.metadata (4.2 kB)Collecting ipykernel==5.5.6 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 10))  Using cached ipykernel-5.5.6-py3-none-any.whl.metadata (1.1 kB)Collecting ipython_genutils==0.2.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 11))  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl.metadata (755 bytes)Collecting jupyter_client==5.2.4 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 12))  Using cached jupyter_client-5.2.4-py2.py3-none-any.whl.metadata (1.5 kB)Collecting jupyter_core==4.6.3 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 13))  Using cached jupyter_core-4.6.3-py2.py3-none-any.whl.metadata (850 bytes)Collecting pexpect==4.8.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 14))  Using cached pexpect-4.8.0-py2.py3-none-any.whl.metadata (2.2 kB)Collecting pickleshare==0.7.5 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 15))  Using cached pickleshare-0.7.5-py2.py3-none-any.whl.metadata (1.5 kB)Collecting ptyprocess==0.7.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 16))  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)Collecting pyzmq==18.0.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req99299714128602899.txt (line 17))  error: subprocess-exited-with-error  Using cached pyzmq-18.0.2.tar.gz (1.2 MB)    × python setup.py egg_info did not run successfully.  Preparing metadata (setup.py): started  Preparing metadata (setup.py): finished with status 'error'  │ exit code: 1  ╰─> [1 lines of output]      error in pyzmq setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '!=3.0*'      [end of output]    note: This error originates from a subprocess, and is likely not a problem with pip.[notice] A new release of pip is available: 24.2 -> 24.3.1[notice] To update, run: /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test4/bin/python -m pip install --upgrade piperror: metadata-generation-failed× Encountered error while generating package metadata.╰─> See above for output.note: This is an issue with the package mentioned above, not pip.hint: See above for details./Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test4/bin/python: No module named ipykernel

/Users/<username>/delete-me/delete-me-venv/bin/python3 (a custom Python venv for Python 3.12.6) throws this error:

Warnings were encountered

    WARNING
    ERR_CODEENV_UPDATE_FAILED
    Environment update failed
    Initial code env packages installation failed
    WARNING
    ERR_CODEENV_JUPYTER_SUPPORT_INSTALL_FAILED
    Jupyter support installation failed
    Cannot install jupyter support

created virtual environment CPython3.12.6.final.0-64 in 319ms  creator CPython3macOsBrew(dest=/Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test5, clear=False, no_vcs_ignore=False, global=False)  seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/Users/edvin.mocibob/Library/Application Support/virtualenv)    added seed packages: pip==24.2  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivatorUpdating code environment according to spec ...Installing from Pip requirementsIgnoring ipykernel: markers 'python_version < "3"' don't match your environmentCollecting pandas<2.3,>=2.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 2))  Using cached pandas-2.2.3-cp312-cp312-macosx_11_0_arm64.whl.metadata (89 kB)Collecting numpy<1.27 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 3))  Using cached numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl.metadata (61 kB)Collecting python-dateutil<3,>=2.8.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 4))  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)Collecting urllib3<2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 5))  Using cached urllib3-1.26.20-py2.py3-none-any.whl.metadata (50 kB)Collecting requests<3 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 6))  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)Collecting decorator==4.4.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 8))  Using cached decorator-4.4.2-py2.py3-none-any.whl.metadata (4.2 kB)Collecting ipykernel==5.5.6 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 10))  Using cached ipykernel-5.5.6-py3-none-any.whl.metadata (1.1 kB)Collecting ipython_genutils==0.2.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 11))  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl.metadata (755 bytes)Collecting jupyter_client==5.2.4 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 12))  Using cached jupyter_client-5.2.4-py2.py3-none-any.whl.metadata (1.5 kB)Collecting jupyter_core==4.6.3 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 13))  Using cached jupyter_core-4.6.3-py2.py3-none-any.whl.metadata (850 bytes)Collecting pexpect==4.8.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 14))  Using cached pexpect-4.8.0-py2.py3-none-any.whl.metadata (2.2 kB)Collecting pickleshare==0.7.5 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 15))  Using cached pickleshare-0.7.5-py2.py3-none-any.whl.metadata (1.5 kB)Collecting ptyprocess==0.7.0 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 16))  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)Collecting pyzmq==18.0.2 (from -r /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/tmp/pip-requirements-install/req11300110537894403721.txt (line 17))  Using cached pyzmq-18.0.2.tar.gz (1.2 MB)  Installing build dependencies: started  Installing build dependencies: finished with status 'done'  Getting requirements to build wheel: started  error: subprocess-exited-with-error    × Getting requirements to build wheel did not run successfully.  │ exit code: 1  ╰─> [1 lines of output]      error in pyzmq setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '!=3.0*'      [end of output]    note: This error originates from a subprocess, and is likely not a problem with pip.  Getting requirements to build wheel: finished with status 'error'[notice] A new release of pip is available: 24.2 -> 24.3.1[notice] To update, run: /Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test5/bin/python -m pip install --upgrade piperror: subprocess-exited-with-error× Getting requirements to build wheel did not run successfully.│ exit code: 1╰─> See above for output.note: This error originates from a subprocess, and is likely not a problem with pip./Users/edvin.mocibob/Library/DataScienceStudio/dss_home/code-envs/python/test5/bin/python: No module named ipykernel

Is there a way to use Python 3.8 or higher for creating code envs in local Dataiku installations?

Operating system used: macOS 15.0.1

Edvin

Thanks for the suggestion Turribeach, but I didn't like the idea of changing system dependencies, so I tried non-managed code environments pointing to Python virtual envs. I still had issues unrelated to my original error (some stuff with the Python packages used in the sample project).

Turribeach

Remove all the installations done via Brew and install all the desired Python versions using the MacOS installers. Then make sure your shell profile adds all the relevant bin directories to the PATH variable (see samples below). Then restart Dataiku and you should be able to create code environments in all the versions you installed. I have 3.8/3.9/3.10/3.11 all working in my Mac DSS installation without any issues.

# Setting PATH for Python 3.8
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
export PATH

# Setting PATH for Python 3.9
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.9/bin:${PATH}"
export PATH

# Setting PATH for Python 3.10
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.10/bin:${PATH}"
export PATH

# Setting PATH for Python 3.11
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.11/bin:${PATH}"
export PATH

Turribeach

This is not changing system dependencies. The correct and official way to install Python on MacOS is to use the python.org MacOS installers. You are having these issues because you are using brew to install Python. The Python MacOS installer can handle multiple Python versions correctly.