New to Dataiku DSS? Try out our NEW Quick Start Programs today and get onboarded on the product in just one hour! Let's go

Does Multiprocessing work on DSS ?

Mass
Level 1
Does Multiprocessing work on DSS ?

Hi !!

I'm using DSS (Version 5.1.4) and want to know if multiprocessing works with a python recipe.

I tried to process a pandas dataframe using packages like multiprocessing, swifter or pandarallel (2, 4 or 8 core) but it doesn't change the processing speed (DSS is installed on a 8 core machine).  

Is there a specific way (or setting) to apply parallel processing on pandas dataframe ?

Thank you !! 

0 Kudos
1 Reply
AlexandreL
Dataiker
Dataiker
Hi,
 
There is no specific way to call multiprocess in a Python recipe within DSS. Please note that python code being ran in DSS is the same as python being ran outside of DSS, we don't do anything special to it.
 
Here are the links that I would recommend looking into with regards to this matter:
0 Kudos
A banner prompting to get Dataiku DSS