Spark on local machine (where DSS is intalled) + Spark on another cluster

UserBird
UserBird Dataiker, Alpha Tester Posts: 535 Dataiker
in Setup & Configuration
Is it possible to configure spark on Dss in a way that we can choose "run Spark on the machine of Dss (local macine)"or run the spark-job with a Spark which is installed on another cluster?

Additionally: How do we configure that Dss interacts with Spark on another cluster
Tagged:
· Share on FacebookShare on Twitter

Answers

  • AdrienL
    AdrienL Dataiker, Alpha Tester Posts: 196 Dataiker

    DSS only supports one Spark installation (i.e. directory with the Spark configuration and libraries for a given version of spark) see here for the details.

    However, using different Spark configurations you can have your Spark job run either locally (spark master "local[*]") or remotely (e.g. spark master "yarn-client" to run on the Hadoop cluster, or spawn a spark "standalone cluster" on your cluster and provide its URL as the spark master).

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…