AWS EC2 instance change inquiry

DAHYE

Hello, I am currently using Dataiku on an m5.4xlarge instance on EC2 on AWS. I need to train a deep learning model (YOLOv8n), so I'm going to use a g4dn instance.
Is it possible to use a g4dn instance only during the training process and use an m5.4xlarge instance for other tasks such as preprocessing or model evaluation?

Also, I would like to know what additional work is required if possible and if not possible.

Turribeach

This is indeed possible when you use containerised execution (ie Kubernetes). This is not trivial but certainly possible.

https://doc.dataiku.com/dss/latest/containers/index.html

DAHYE

Thank you

I don't have much time. Is this possible without Kubernetes? In the current state, the instance appears to change if you only change the instance type option of AWS. Would it be impossible to use the g4dn instance only for training by changing the option?

If I proceed without Kubernetes, do I need to deploy dataiku again if I change the instance?

Turribeach

You can change the instance type but this requires the instance to be down. In some cases AWS will not allow you to change the instance type or the instance may not boot. If you can change the instance type you won't need to reinstall Dataiku but GPU training is usually a complex setup. A few years ago I posted about the steps required to get GPU training working in Dataiku. This post is obviously too old now as it's based on RHEL 7.9 which Dataiku no longer supports. So you will need to come up with your own set of steps and compatible driver versions to get this working according to your particular GPU and OS versions. The Dataiku GPU requirements are here:

https://doc.dataiku.com/dss/latest/machine-learning/computer-vision/runtime-gpu.html#requirements

DAHYE

Thank you for your kind reply.
I will proceed by referring to the attached Link.
Thank you so much!