Deep Neural Network Runtime Environment

Stefan1
Stefan1 Partner, Dataiku DSS Core Designer, Registered Posts: 3 Partner

Hi
I installed Dataiku on Azure. The Deep Neural Network requires a Runtime Environment. I installed it based on https://doc.dataiku.com/dss/latest/machine-learning/deep-learning/first-model.html#create-a-code-environment-with-the-required-packages . However the proposed “Visual Deep Learning: Tensorflow. CPU, and GPU with CUDA11.2 + cuDNN 8.1” did not exist, so i took "Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1)".

However in the Visual Analysis the Runtime environment still has a failure.1.png

As it mentioned that torch and skorch are missing, i installed them in the runtime environment as well.

2.png

Can you help me to use the deep neural network on a normal dataikue azure installation?

Regards

Stefan


Operating system used: Azure

Tagged:

Answers

  • AlexandreV
    AlexandreV Dataiker, Registered Posts: 28 Dataiker

    Hello @Stefan1
    ,

    In Dataiku DSS you can create neural networks from both the Visual Deep Learning and the Visual Machine Learning modeling tasks.

    The documentation you are following is referring to the Visual Deep Learning models using Tensorflow, whereas the code env packages are related to the Visual Machine Learning modeling task with the deep learning algorithm.

    To solve your issue and use the deep neural network algorithm of the Visual Machine Learning modeling task, you need to use the Visual Machine Learning with deep neural network (GPU - CUDA 11.1) package preset in the code env settings.

    Alternatively, if you want to use the Visual Deep Learning model, you can keep the code env you created (no need to add skorch and pytorch)

    Visual Deep Learning

    For this you need to use the Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1) code env (the one you created)

    Screenshot 2023-08-25 at 17.10.35.png

    Screenshot 2023-08-25 at 17.11.43.png

    Screenshot 2023-08-25 at 17.21.21.png

    Visual Machine Learning with deep neural network algorithm

    On the other hand there is a Deep Neural Network algorithm. This one will need a code env with Visual Machine Learning with deep neural network (GPU - CUDA 11.1) packages preset.

    Screenshot 2023-08-25 at 17.19.32.png

    Screenshot 2023-08-25 at 17.15.59.png

    Screenshot 2023-08-25 at 17.21.42.png

    Hope it helps,

    Alex

  • Stefan1
    Stefan1 Partner, Dataiku DSS Core Designer, Registered Posts: 3 Partner

    Hi Alex

    I intent to use the "Visual Machine Learning with deep neural network algorithm". I Installed the "Visual Machine Learning with deep neural network (GPU - CUDA 11.1) " howerver Dataiku ran into a problem with the code env:

    3e.png

    Do you know the reason?

    Regards Stefan

  • AlexandreV
    AlexandreV Dataiker, Registered Posts: 28 Dataiker

    Hello @Stefan1
    ,

    You are creating the right code env.
    I am not sure what is going wrong here.

    Could you send the updateEnvAccordingToSpec.log and give me a few info about your setup?
    - OS
    - DSS version

    Screenshot 2023-08-28 at 10.23.57.png

    Regards,
    Alex

  • Stefan1
    Stefan1 Partner, Dataiku DSS Core Designer, Registered Posts: 3 Partner

    Hi Alex
    I send you the logs.

    I installed Dataiku on Azure following this instructions: https://www.dataiku.com/product/get-started/azure/. It is a rather fresh installation from last week. The DSS version is: 12.1.0

    Regards Stefan

  • AlexandreV
    AlexandreV Dataiker, Registered Posts: 28 Dataiker

    Hi @Stefan1
    ,

    I did reproduce your error on a D2 vm on Azure: the error is that the machine has not enough memory to run the install of pytorch.

    I have reported this error on the documentation : we should advise the use of a bigger machine around 16GB of memory (you can see the requirements for linux installs).

    The real fix is to change the machine type from D2 to a bigger one (or reinstall dss on a new VM) and that's what I would advise you to do.

    However, there is a workaround to create your code env : you can add a --no-cache-dir extra option for pip

    Screenshot 2023-08-28 at 15.09.42.png

     This will allow you to create the code env.

    Regards,

    Alex

Setup Info
    Tags
      Help me…