Survey banner
The Dataiku Community is moving to a new home! We are temporary in read only mode: LEARN MORE

Deep Neural Network Runtime Environment

Stefan1
Level 2
Deep Neural Network Runtime Environment

Hi 
I installed Dataiku on Azure. The Deep Neural Network requires a Runtime Environment. I installed it based on https://doc.dataiku.com/dss/latest/machine-learning/deep-learning/first-model.html#create-a-code-env... . However the proposed  “Visual Deep Learning: Tensorflow. CPU, and GPU with CUDA11.2 + cuDNN 8.1” did not exist, so i took "Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1)".

However in the Visual Analysis the Runtime environment still has a failure.1.png

As it mentioned that torch and skorch are missing, i installed them in the runtime environment as well. 

2.png

Can you help me to use the deep neural network on a normal dataikue azure installation?

 

Regards

Stefan


Operating system used: Azure

0 Kudos
5 Replies
AlexandreV
Dataiker

Hello @Stefan1 ,

In Dataiku DSS you can create neural networks from both the Visual Deep Learning and the Visual Machine Learning modeling tasks.

The documentation you are following is referring to the Visual Deep Learning models using Tensorflow, whereas the code env packages are related to the Visual Machine Learning modeling task with the deep learning algorithm.

To solve your issue and use the deep neural network algorithm of the Visual Machine Learning modeling task, you need to use the Visual Machine Learning with deep neural network (GPU - CUDA 11.1) package preset in the code env settings.

Alternatively, if you want to use the Visual Deep Learning model, you can keep the code env you created (no need to add skorch and pytorch)

Visual Deep Learning

For this you need to use the Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1) code env (the one you created)

Screenshot 2023-08-25 at 17.10.35.png

Screenshot 2023-08-25 at 17.11.43.png

Screenshot 2023-08-25 at 17.21.21.png

Visual Machine Learning with deep neural network algorithm

On the other hand there is a Deep Neural Network algorithm. This one will need a code env with Visual Machine Learning with deep neural network (GPU - CUDA 11.1) packages preset.

Screenshot 2023-08-25 at 17.19.32.png

Screenshot 2023-08-25 at 17.15.59.png

Screenshot 2023-08-25 at 17.21.42.png

 

Hope it helps,

Alex

0 Kudos
Stefan1
Level 2
Author

Hi Alex

I intent to use the "Visual Machine Learning with deep neural network algorithm". I Installed the "Visual Machine Learning with deep neural network (GPU - CUDA 11.1) " howerver Dataiku ran into a problem with the code env:

3e.png

Do you know the reason?

Regards Stefan

0 Kudos
AlexandreV
Dataiker

Hello @Stefan1,

You are creating the right code env.
I am not sure what is going wrong here.

Could you send the updateEnvAccordingToSpec.log and give me a few info about your setup?
- OS
- DSS version

Screenshot 2023-08-28 at 10.23.57.png

Regards,
Alex

0 Kudos
Stefan1
Level 2
Author

Hi Alex
I send you the logs. 

I installed Dataiku on Azure following this instructions: https://www.dataiku.com/product/get-started/azure/. It is a rather fresh installation from last week. The DSS version is: 12.1.0

Regards Stefan

0 Kudos
AlexandreV
Dataiker

Hi @Stefan1,

I did reproduce your error on a D2 vm on Azure: the error is that the machine has not enough memory to run the install of pytorch.

I have reported this error on the documentation : we should advise the use of a bigger machine around 16GB of memory (you can see the requirements for linux installs).

The real fix is to change the machine type from D2 to a bigger one (or reinstall dss on a new VM) and that's what I would advise you to do.

However, there is a workaround to create your code env : you can add a --no-cache-dir extra option for pip

Screenshot 2023-08-28 at 15.09.42.png

 This will allow you to create the code env.

 

Regards,

Alex