Deep Neural Network Runtime Environment

Stefan1

Hi
I installed Dataiku on Azure. The Deep Neural Network requires a Runtime Environment. I installed it based on https://doc.dataiku.com/dss/latest/machine-learning/deep-learning/first-model.html#create-a-code-environment-with-the-required-packages . However the proposed “Visual Deep Learning: Tensorflow. CPU, and GPU with CUDA11.2 + cuDNN 8.1” did not exist, so i took "Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1)".

However in the Visual Analysis the Runtime environment still has a failure.

As it mentioned that torch and skorch are missing, i installed them in the runtime environment as well.

Can you help me to use the deep neural network on a normal dataikue azure installation?

Regards

Stefan

---

Operating system used: Azure

AlexandreV

Hello @Stefan1
,

In Dataiku DSS you can create neural networks from both the Visual Deep Learning and the Visual Machine Learning modeling tasks.

The documentation you are following is referring to the Visual Deep Learning models using Tensorflow, whereas the code env packages are related to the Visual Machine Learning modeling task with the deep learning algorithm.

To solve your issue and use the deep neural network algorithm of the Visual Machine Learning modeling task, you need to use the Visual Machine Learning with deep neural network (GPU - CUDA 11.1) package preset in the code env settings.

Alternatively, if you want to use the Visual Deep Learning model, you can keep the code env you created (no need to add skorch and pytorch)

Visual Deep Learning

For this you need to use the Visual Deep Learning: Tensorflow (GPU - CUDA 11.2 / cuDNN 8.1) code env (the one you created)

 

Visual Machine Learning with deep neural network algorithm

On the other hand there is a Deep Neural Network algorithm. This one will need a code env with Visual Machine Learning with deep neural network (GPU - CUDA 11.1) packages preset.

Hope it helps,

Alex

Stefan1

Hi Alex

I intent to use the "Visual Machine Learning with deep neural network algorithm". I Installed the "Visual Machine Learning with deep neural network (GPU - CUDA 11.1) " howerver Dataiku ran into a problem with the code env:

Do you know the reason?

Regards Stefan

AlexandreV

Hello @Stefan1
,

You are creating the right code env.
I am not sure what is going wrong here.

Could you send the updateEnvAccordingToSpec.log and give me a few info about your setup?
- OS
- DSS version

 

Regards,
Alex

Stefan1

Hi Alex
I send you the logs.

I installed Dataiku on Azure following this instructions: https://www.dataiku.com/product/get-started/azure/. It is a rather fresh installation from last week. The DSS version is: 12.1.0

Regards Stefan

AlexandreV

Hi @Stefan1
,

I did reproduce your error on a D2 vm on Azure: the error is that the machine has not enough memory to run the install of pytorch.

I have reported this error on the documentation : we should advise the use of a bigger machine around 16GB of memory (you can see the requirements for linux installs).

The real fix is to change the machine type from D2 to a bigger one (or reinstall dss on a new VM) and that's what I would advise you to do.

However, there is a workaround to create your code env : you can add a --no-cache-dir extra option for pip

 This will allow you to create the code env.

Regards,

Alex