Want to Stop Rebuilding "Expensive" Parts of your Flow? Explicit Builds are the Answer!READ MORE

Regarding AUto ml session

akash97715
Level 1
Regarding AUto ml session

Hi Team,

While training the model with auto ML. there are 6-7 algorithm job starts in a session. Is there a way all algorithm should start at a time? Or this is something what dataiku automatically decides which algorithm to start to achieve optimal memory execution.

 

In the attached screenshot we see Logistic regression training started and SGD is in pending state. So the question here is is there any configuration or setting which will start all the algorithm at same time, or this is something dataiku automatically manages to achieve optimal memory execution.

 

Auto ML Training.PNG

0 Kudos
1 Reply
tgb417
Neuron
Neuron

@akash97715 

Welcome to the Dataiku community.

Auto ML jobs  are usually computationally complex.  Depending on how much computational power the computer you are running dss on has, you may not want to increase the number of concurrent jobs. This may actually slow things down because you don’t have enough CPU cores or RAM memory.  Note the resources we are talking about are not your local computer but the server(s) on which DSS is running.  

I’d take a look at top, or htop or one of the other tasks manager on the computer running these jobs to see if you have significant unused system resources.  This could be more complicated to evaluate if you are doing your compute on kubernetes.

If you do find that you have a bunch of unused resources you might consider increasing the concurrent limits.  See the documentation pages.

https://doc.dataiku.com/dss/latest/flow/limits.html 

--Tom
0 Kudos