Spark Cluster mode

Bader
Bader Registered Posts: 46 ✭✭✭✭✭

Hello,

As we are using Spark heavily, we are having the problem of slowness of application launching in yarn cluster mode. The slowness comes from having many DSS related files and also many jars files has to be uploaded for every single spark application.

We checked the feature of using Cluster mode. However, we know that is not supported.

We need a guide of how to use it even if its not supported.

Thanks

Tagged:

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker

    Hi,
    I believe this has been discussed directly with your Dataiku Account team. There is currently no way to run in cluster mode and running in cluster would not make it faster as the same Jars would still need to be loaded.

Setup Info
    Tags
      Help me…