Support for distributed/multi node grid search/training? python/sci-kit learn based

UserBird Dataiker
Dataiker
Support for distributed/multi node grid search/training? python/sci-kit learn based
Jump to solution
Hi, what is the support in Dataiku for distributing grid searches/training? Not speaking of spark or hadoop existing dataiku integration, but mainly sending the regular python based jobs to other nodes, that could be running dask or similar frameworks, get the results back to dataiku transparently? possible? if so, what is the setup/licensing needed? Could it be done installing additional dataiku servers on nodes?

thx!
0 Kudos
1 Solution

Accepted Solutions
Highlighted
Alex_Combessie Dataiker
Dataiker
Re: Support for distributed/multi node grid search/training? python/sci-kit learn based
Jump to solution
Hi Rui,

Note that we already distribute training on CPUs. Distributed training is something we are working on at the moment. It will be part of one of our next software releases. We will keep you posted on the details when it is ready.

In the meantime, you can develop it with custom code in Python/Scala/Shell/R scripts within Dataiku DSS.

Cheers,

Alexandre

View solution in original post

0 Kudos
2 Replies
Highlighted
Alex_Combessie Dataiker
Dataiker
Re: Support for distributed/multi node grid search/training? python/sci-kit learn based
Jump to solution
Hi Rui,

Note that we already distribute training on CPUs. Distributed training is something we are working on at the moment. It will be part of one of our next software releases. We will keep you posted on the details when it is ready.

In the meantime, you can develop it with custom code in Python/Scala/Shell/R scripts within Dataiku DSS.

Cheers,

Alexandre

View solution in original post

0 Kudos
Highlighted
UserBird Dataiker
Dataiker
Author
Re: Support for distributed/multi node grid search/training? python/sci-kit learn based
Jump to solution
Thx Alexandre, yes, perfectly clear on the cpus, very easy to notice.
Awesome news, tanks for the heads-up, think this is really mandatory these days, otherwise very limiting comparing to dask & similar, which are opensource/free.
regards
Rui
0 Kudos
Labels (3)