Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Trying to use the spark engine, configured to a spark cluster in kubernetes.
Using scala or spark-submit works well.
Some visual recipes like "Analyze" work well too.
Other visual recipes like "GroupBy" or "Pivot" give me the same error. Does anybody know what it means? or what it is caused by? I'm using dataiku 9.0 and spark 3.0.0.
Hi,
the error implies that the jars on the images used by the kubernetes pods and the jars in the spark-submit called by DSS are different. You should make sure the setup is consistent:
- rerun dssadmin install-spark-integration ...
- then rebuild the spark image with dssadmin build-base-image --type spark ...
- and push the base images to the cloud repository with the "push base images" button on the Administration > Settings > Spark tab.
If the problem persists, you should open a support ticket with a full diagnostic of the failed job.
Thank you, I used my own spark image and it worked properly for a while. I'll rebuild the image and tell you how that goes.