Are there different ways to set up code environments?
Options
Graham_E
Registered Posts: 14 ✭✭✭✭
I am trying to install pytorch in python3 in a code environment in data science studio. I can install it in the python3.5 install on the system that Data Science Studio is installed on. I've tried putting torch in the REQUESTED PACKAGES (PIP) part of the code environment administration but that doesn't work because pytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it (instructions are here: http://pytorch.org/).
I've noticed there is a TODO section coming in the documentation (https://doc.dataiku.com/dss/latest/code-envs/operations-python.html) which probably gives more details about this. I will also be curious how to install a custom project (from git) in a way that it can be used in a custom code environment. FYI I have been able to install both pytorch and the custom project I am alluding to for use with the system install of python3.5 on the server I have DSS installed on.
I'm not sure if my description of the problem is clear so please ask questions if there are any ambiguities. Thanks!
I've noticed there is a TODO section coming in the documentation (https://doc.dataiku.com/dss/latest/code-envs/operations-python.html) which probably gives more details about this. I will also be curious how to install a custom project (from git) in a way that it can be used in a custom code environment. FYI I have been able to install both pytorch and the custom project I am alluding to for use with the system install of python3.5 on the server I have DSS installed on.
I'm not sure if my description of the problem is clear so please ask questions if there are any ambiguities. Thanks!
Best Answer
-
Hi,
The syntax in the "requested packages" is actually a pip requirements file, so you can simply put the URL of your wheel here
So in essence, you just need the following in your "requested packages"
http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
torchvision
Answers
-
Awesome, thanks! I'm also curious about how to install a package that isn't done through pip. In the docs you have:
TODO: Explain that admin first downloads them and then how to put that in requirements.txt
I'm trying to install this if it helps: https://github.com/huggingface/torchMoji. The install command I use is:
pip3 install -e . -
Clone the repository on the DSS machine as you would normally do, then add the following line to the "Requested packages":
-e /absolute/path/to/where/you/cloned/torchMoji -
MarkPundurs Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 26 ✭✭✭✭Optionspytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it
As I understand it, this is incorrect as it stands; Pytorch requires you to download a pip wheel for installing it with CUDA support.
And in 12.1 it may be the case that no special handling is required; I simply entered torch in the "Requested packages" list and had the following packages (among others) installed:
nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu12==12.1.105