Dataiku Spark Remote Connection
Hello, I am using Dataiku 12.5.2 and currently running Spark 2.4. Dataiku is installed on a server named A, while Spark is installed on a server named B, configured as a standalone Spark installation without Hadoop. Both server A and server B are capable of TCP communication and allow SSH access. How can I use the Spark on server B from Dataiku on server A?
※ I have confirmed that the Spark installed on server A, where Dataiku is also installed, is well integrated.
@spark
Operating system used: linux
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,161 NeuronPlease review the different Spark integration options in the documentation:
https://doc.dataiku.com/dss/latest/spark/installation.html#setting-up-spark-integration
-
SangHoon Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 15 ✭✭✭
I am using Dataiku and Spark in an on-premise environment. Spark is configured as a standalone installation without Hadoop. Due to the current project configuration, I cannot use Kubernetes (k8s) or Docker. I would like to integrate Dataiku on server A with Spark on server B.
-
Did you solve the problem and how ?
