Spark Cluster mode

Bader

Hello,

As we are using Spark heavily, we are having the problem of slowness of application launching in yarn cluster mode. The slowness comes from having many DSS related files and also many jars files has to be uploaded for every single spark application.

We checked the feature of using Cluster mode. However, we know that is not supported.

We need a guide of how to use it even if its not supported.

Thanks

Alexandru

Hi,
I believe this has been discussed directly with your Dataiku Account team. There is currently no way to run in cluster mode and running in cluster would not make it faster as the same Jars would still need to be loaded.