Model training using Azure Databricks - Doesn't work in Dataiku DSS Enterprise Edition Trial
Model training using Azure Databricks doesn’t work. We have used the DSS VM by following the instruction https://www.dataiku.com/product/get-started/azure/ looks like this instance doesn’t have already necessary libraries for spark-submit and many other dependencies.
1) Configured the Databricks setup under Administration > Settings > Spark with Access Token etc.,
2) Getting exception Cannot run program "spark-submit" (in directory "/home/dataiku/dss/tmp/sparkbased-doctor/out8029364253325165783"): error=2, No such file or directory, caused by: IOException: error=2, No such file or directory when model training started.
Leveraging Databricks for Spark execution is a very specific setup that requires a dedicated procedure and involvement with Dataiku tech resources. There may be better suited alternatives depending on what exactly you are trying to accomplish.
Just like your other post, we would strongly encourage you to get in touch with us (https://www.dataiku.com/home/contact-us/), so that a Dataiku representative can better understand your needs, so as to set you up as well as possible on evaluation of specific technologies and capabilities.