Model training using Azure Databricks - Doesn't work in Dataiku DSS Enterprise Edition Trial
Hi,
Model training using Azure Databricks doesn’t work. We have used the DSS VM by following the instruction https://www.dataiku.com/product/get-started/azure/ looks like this instance doesn’t have already necessary libraries for spark-submit and many other dependencies.
Steps done:
1) Configured the Databricks setup under Administration > Settings > Spark with Access Token etc.,
2) Getting exception Cannot run program "spark-submit" (in directory "/home/dataiku/dss/tmp/sparkbased-doctor/out8029364253325165783"): error=2, No such file or directory, caused by: IOException: error=2, No such file or directory when model training started.
Additional Info:
1) I use Dataiku DSS Enterprise Edition Trial
2) The DSS Instance is created in Azure by following https://www.dataiku.com/product/get-started/azure/
3) The DSS instance has the license file.
4) Though the License says its DSS Enterprise Edition (Trial) but still when I use the License in my DSS Admin page it shows me as “Free Edition (with advanced features trial)”.
Can you please help on this issue.
Answers
-
Hi,
Leveraging Databricks for Spark execution is a very specific setup that requires a dedicated procedure and involvement with Dataiku tech resources. There may be better suited alternatives depending on what exactly you are trying to accomplish.
Just like your other post, we would strongly encourage you to get in touch with us (https://www.dataiku.com/home/contact-us/), so that a Dataiku representative can better understand your needs, so as to set you up as well as possible on evaluation of specific technologies and capabilities.
