Are there different ways to set up code environments?

Graham_E
Graham_E Registered Posts: 14 ✭✭✭✭
I am trying to install pytorch in python3 in a code environment in data science studio. I can install it in the python3.5 install on the system that Data Science Studio is installed on. I've tried putting torch in the REQUESTED PACKAGES (PIP) part of the code environment administration but that doesn't work because pytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it (instructions are here: http://pytorch.org/).

I've noticed there is a TODO section coming in the documentation (https://doc.dataiku.com/dss/latest/code-envs/operations-python.html) which probably gives more details about this. I will also be curious how to install a custom project (from git) in a way that it can be used in a custom code environment. FYI I have been able to install both pytorch and the custom project I am alluding to for use with the system install of python3.5 on the server I have DSS installed on.

I'm not sure if my description of the problem is clear so please ask questions if there are any ambiguities. Thanks!

Best Answer

Answers

  • Graham_E
    Graham_E Registered Posts: 14 ✭✭✭✭
    Awesome, thanks! I'm also curious about how to install a package that isn't done through pip. In the docs you have:

    TODO: Explain that admin first downloads them and then how to put that in requirements.txt

    I'm trying to install this if it helps: https://github.com/huggingface/torchMoji. The install command I use is:

    pip3 install -e .
  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Clone the repository on the DSS machine as you would normally do, then add the following line to the "Requested packages":

    -e /absolute/path/to/where/you/cloned/torchMoji
  • MarkPundurs
    MarkPundurs Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 27 ✭✭✭✭
    edited July 17
    pytorch can't be installed directly through pip. Pytorch requires you to download a pip wheel for installing it

    As I understand it, this is incorrect as it stands; Pytorch requires you to download a pip wheel for installing it with CUDA support.

    And in 12.1 it may be the case that no special handling is required; I simply entered torch in the "Requested packages" list and had the following packages (among others) installed:

    nvidia-cuda-cupti-cu12==12.1.105
    nvidia-cuda-nvrtc-cu12==12.1.105
    nvidia-cuda-runtime-cu12==12.1.105

Setup Info
    Tags
      Help me…