Timeseries forecasting with GPU / cuda 11

raftersvk
raftersvk Registered Posts: 6 ✭
in Using Dataiku

Hello,

I am now trying to train a model with timeseries forecast by using GPU.
OS: Ubuntu 22.04

Installed with apt-get on OS:

  • libcudnn9-cuda-11
  • cuda-toolkit-11-8
  • libnccl2

I then created a new python env :

image.png

when i use that environment in the model, I can see at first that it's fine since it shows me my GPU card :

image.png

but when I start training there is an error :

image.png



and the result when I try to execute a ML model :

image.png

Any idea how to resolve the issue ?

Operating system used: Ubuntu 22.04

Tagged:
· Share on FacebookShare on Twitter

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,088 Neuron

    You need to add all the additional software and drivers that NVIDIA GPUs need to work. These steps will be of course dependant on the OS / version / architecture / hardware / GPU that your server is running and it's not a trivial setup given that all the software versions need to be compatible between themselves and the GPU used and this not always clearly stated on each of the software components. Pretty much all the cloud vendors provide OS images with the software components pre-installed and configured properly to be used on GPU enabled instances so you may be able to leverage those if you are using a Cloud VM. Over 3 years ago I wrote this post which is a complete guide on all the setup steps needed to get GPU training working in a Dataiku instance running on RHEL v7.9. While this post is now outdated it will give you a rough idea of all the steps involved. It will be up to you to work out the specific steps for your required environment. Feel free to post an update when and if you get it working so other people in the Community can benefit from your experience.

    · Share on FacebookShare on Twitter
  • raftersvk
    raftersvk Registered Posts: 6 ✭
    edited October 9

    I will look into that but my first impression is that TimeSeries Forecasting hasn't been updated for a while.

    Indeed the DL algorithms proposed in the default install are based on the MxNet library which is no longer maintained since November 2023. Also it is stucked with a cuda 11.7 at max.

    this makes things even more complicated since my setup is based on CUDA 12.5 for other reasons / tools.

    any idea if its possible to work with newer version of algorithms ? libraries ?

    · Share on FacebookShare on Twitter
  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,088 Neuron

    DL is an advanced topic which is probably not suitable to test on the free Dataiku version specially if you want the latest frameworks. Ultimately Dataiku has to targe specific framework versions to integrate with them and make sure they work. These frameworks are a moving target so you can't really expect Dataiku to always support the latest versions. Support for Pandas 2.x has only just recently been added for instance. Whether this will work or not it's hard to say. If I had to guess I would probably say CUDA 12.5 will not work but it's just a guess. You will need to involve Dataiku Support or Professional Services to get a more clear answer.

    · Share on FacebookShare on Twitter
  • raftersvk
    raftersvk Registered Posts: 6 ✭

    Thank you for your answer and I will give another try with "generic" AutoML prediction to see if I can make it work.

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…