Regarding AUto ml session

Options
akash97715
akash97715 Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 8 ✭✭✭✭

Hi Team,

While training the model with auto ML. there are 6-7 algorithm job starts in a session. Is there a way all algorithm should start at a time? Or this is something what dataiku automatically decides which algorithm to start to achieve optimal memory execution.

In the attached screenshot we see Logistic regression training started and SGD is in pending state. So the question here is is there any configuration or setting which will start all the algorithm at same time, or this is something dataiku automatically manages to achieve optimal memory execution.

Auto ML Training.PNG

Answers

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @akash97715

    Welcome to the Dataiku community.

    Auto ML jobs are usually computationally complex. Depending on how much computational power the computer you are running dss on has, you may not want to increase the number of concurrent jobs. This may actually slow things down because you don’t have enough CPU cores or RAM memory. Note the resources we are talking about are not your local computer but the server(s) on which DSS is running.

    I’d take a look at top, or htop or one of the other tasks manager on the computer running these jobs to see if you have significant unused system resources. This could be more complicated to evaluate if you are doing your compute on kubernetes.

    If you do find that you have a bunch of unused resources you might consider increasing the concurrent limits. See the documentation pages.

    https://doc.dataiku.com/dss/latest/flow/limits.html

Setup Info
    Tags
      Help me…