Check out the first Dataiku 8 Deep Dive focusing on Productivity on October 29th Read More

Does Multiprocessing work on DSS ?

Level 1
Does Multiprocessing work on DSS ?

Hi !!

I'm using DSS (Version 5.1.4) and want to know if multiprocessing works with a python recipe.

I tried to process a pandas dataframe using packages like multiprocessing, swifter or pandarallel (2, 4 or 8 core) but it doesn't change the processing speed (DSS is installed on a 8 core machine).  

Is there a specific way (or setting) to apply parallel processing on pandas dataframe ?

Thank you !! 

0 Kudos
1 Reply
Dataiker
Dataiker
Hi,
 
There is no specific way to call multiprocess in a Python recipe within DSS. Please note that python code being ran in DSS is the same as python being ran outside of DSS, we don't do anything special to it.
 
Here are the links that I would recommend looking into with regards to this matter:
0 Kudos