Configuring Spark and Hadoop with Dataiku DSS
Hi! I'm encountering difficulties configuring Spark and Hadoop with Dataiku DSS on my local Windows machine. I've successfully installed Dataiku DSS (Free Edition with advanced features trial), Spark (version 3.5.1), and Hadoop (version 3.3.0) on my system. However, I'm unable to configure Spark for use within Dataiku DSS, and I also aim to configure Hadoop to allow dataset uploads from HDFS.
The issue occurs within the setup and configuration process of Dataiku DSS on my local Windows machine.
What version are you using?
Dataiku DSS: Free Edition (with advanced features trial)
Spark: 3.5.1
Hadoop: 3.3.0
I've searched online for configuration guides and tried various configurations, but I haven't been successful in getting Spark and Hadoop properly configured within Dataiku DSS.
Operating system used: Windows
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptionsHi, as per the Dataiku Windows page the Dataiku Windows download is provided for experimental purposes and it's unsupported. If you want to fully test all the features you will need to be in a supported configuration. The VM solution is probably best for you if you are in Windows.
-
Thanks, I'm now on the VM. How can I configure Spark and Hadoop?
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptions