Are there different ways to set up code environments?

Solved!
Graham_E
Level 2
Are there different ways to set up code environments?
I am trying to install pytorch in python3 in a code environment in data science studio. I can install it in the python3.5 install on the system that Data Science Studio is installed on. I've tried putting torch in the REQUESTED PACKAGES (PIP) part of the code environment administration but that doesn't work because pytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it (instructions are here: http://pytorch.org/).

I've noticed there is a TODO section coming in the documentation (https://doc.dataiku.com/dss/latest/code-envs/operations-python.html) which probably gives more details about this. I will also be curious how to install a custom project (from git) in a way that it can be used in a custom code environment. FYI I have been able to install both pytorch and the custom project I am alluding to for use with the system install of python3.5 on the server I have DSS installed on.

I'm not sure if my description of the problem is clear so please ask questions if there are any ambiguities. Thanks!
0 Kudos
1 Solution
Clรฉment_Stenac
Hi,

The syntax in the "requested packages" is actually a pip requirements file, so you can simply put the URL of your wheel here

So in essence, you just need the following in your "requested packages"

http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
torchvision

View solution in original post

4 Replies
Clรฉment_Stenac
Hi,

The syntax in the "requested packages" is actually a pip requirements file, so you can simply put the URL of your wheel here

So in essence, you just need the following in your "requested packages"

http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
torchvision
Graham_E
Level 2
Author
Awesome, thanks! I'm also curious about how to install a package that isn't done through pip. In the docs you have:

TODO: Explain that admin first downloads them and then how to put that in requirements.txt

I'm trying to install this if it helps: https://github.com/huggingface/torchMoji. The install command I use is:

pip3 install -e .
0 Kudos
Clรฉment_Stenac
Clone the repository on the DSS machine as you would normally do, then add the following line to the "Requested packages":

-e /absolute/path/to/where/you/cloned/torchMoji
0 Kudos
MarkPundurs
Level 3
pytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it

As I understand it, this is incorrect as it stands; Pytorch requires you to download a pip wheel for installing it with CUDA support.

And in 12.1 it may be the case that no special handling is required; I simply entered torch in the "Requested packages" list and had the following packages (among others) installed:

nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105

 

0 Kudos