DSS not finding python.exe with conda

[C130PA5]

Hi,

I am trying to setup DSS 12 to use conda (python3.9) on windows.

I included conda to my path. However, when setting a new code environment Managed by DSS using conda, the environment creation fails after initialisation and package download with the following error:

"<path_to_env>\Scripts\python.exe": CreateProcess error=2, The system cannot find the file specified, caused by: IOException: CreateProcess error=2, The system cannot find the file specified

After looking around and trying both conda and miniconda, it looks like DSS is configured to look for the python executable in the scripts folder meanwhile both conda distributions package the python.exe in the root folder.

Thanks

---

Operating system used: Windows 11

---

Operating system used: Windows 11

[Turribeach]

My advice will be that you stick a standard Python installation from Python.org, select the option of adding Python to the PATH variable, then restart Dataiku and use the new Python version via pip. 

[C130PA5]

@Turribeach, I do have a standard environment already working. The problem with that is the use of libraries with prepackaged executable that are not well supported by pip.

Since conda support is implemented into the software, the issue at hand should be fixed. Otherwise, why support it?

[Turribeach]

Irrespective of you having a virtual environment already working, you will need to recreate it on Dataiku, either using pip or conda. The main advantage of using conda (ignoring the licensing issues of course) over pip is that it handles OS level dependencies. For someone working outside Dataiku that may be appealing since you can get conda to do manage both OS level stuff and Python packages.

With Dataiku I think this is more of a curse than an advantage. The last thing you want is some random user installing an OS level package that makes your system unstable. That's why pip is much preferred so users can only install Python packages and if any OS level packages are required then you will sort it manually having full control of what gets added to the machine. 

Since you mention you are running Dataiku on Windows I presume you are just playing with Dataiku and this is not a production machine (since Dataiku is not officially supported on Windows), but I thought I would mention it anyway. 

So now comes my question. Are you trying to add an existing code environment or reusing the base Python installation that came with your conda install? 

[C130PA5]

@TurribeachI have to specify, I'm a scientist working with DSS on a personal machine and not a developer looking to deploy DSS. So yes, I was just playing around with DSS and trying to see if I can use it to centralize my data analysis python scripts. Also, when I said I have a running environment, I meant python environment in DSS.

As I mentionned in my initial post, I'm trying to add a new code env to DSS with conda support which would create a new conda environment dedicated for DSS. When DSS tries to initialize this new environment, it copies the conda directory structure where the python.exe is not inside the Scripts folder, but in the root directory of the environment. However, for no reason DSS still looks for the executable in the scripts folder expecting it to be there.

[Turribeach]

I have yet to find the need to use conda for my Dataiku code environments. If you were creating code environments on Kubernetes for instance conda could be a better option. But inside Dataiku pip is the better option. Since you are just playing around I will suggest you use pip and create a new pip code environment.