Spark Cluster mode
Bader
Registered Posts: 46 ✭✭✭✭✭
Hello,
As we are using Spark heavily, we are having the problem of slowness of application launching in yarn cluster mode. The slowness comes from having many DSS related files and also many jars files has to be uploaded for every single spark application.
We checked the feature of using Cluster mode. However, we know that is not supported.
We need a guide of how to use it even if its not supported.
Thanks
Tagged:
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker
Hi,
I believe this has been discussed directly with your Dataiku Account team. There is currently no way to run in cluster mode and running in cluster would not make it faster as the same Jars would still need to be loaded.