Spark job error: NoSuchMethodError

Solved!
ismail
Level 2
Spark job error: NoSuchMethodError

Hi,

May I know if my error has got to do with the spark version?

Please advise what is the way forward? Thanks!

 

  • DSS version: 7.0.2
  • Spark version in my Cloudera 6.3.2 CDH cluster: 2.4.0
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.854] [main] [INFO] [org.apache.spark.SparkContext]  - Created broadcast 0 from newAPIHadoopRDD at HDFSDatasetRDD.scala:160
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.928] [main] [ERROR] [dku.spark.env]  - job failed
[12:24:08] [INFO] [dku.utils]  - java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/DataFrame;
[12:24:08] [INFO] [dku.utils]  - 	at com.dataiku.dip.spark.StdDataikuSparkContext.getAsHadoopRDDDataFrameFromHDFS(StdDataikuSparkContext.scala:476)
[12:24:08] [INFO] [dku.utils]  - 	at com.dataiku.dip.spark.StdDataikuSparkContext.getAsSampledDataFrameInternal(StdDataikuSparkContext.scala:197)
[12:24:08] [INFO] [dku.utils]  - 	at com.dataiku.dip.spark.StdDataikuSparkContext.getAsSampledDataFrame(StdDataikuSparkContext.scala:145)
[12:24:08] [INFO] [dku.utils]  - 	at com.dataiku.dip.shaker.sparkimpl.ShakerSparkEntryPoint$.main(ShakerSparkEntryPoint.scala:37)
[12:24:08] [INFO] [dku.utils]  - 	at com.dataiku.dip.shaker.sparkimpl.ShakerSparkEntryPoint.main(ShakerSparkEntryPoint.scala)
[12:24:08] [INFO] [dku.utils]  - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[12:24:08] [INFO] [dku.utils]  - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[12:24:08] [INFO] [dku.utils]  - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[12:24:08] [INFO] [dku.utils]  - 	at java.lang.reflect.Method.invoke(Method.java:498)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
[12:24:08] [INFO] [dku.utils]  - 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint]  - Spark: initializeApplication
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint]  - Spark: initializeApplication: ALREADY done
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint]  - Loading exec env
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.945] [main] [INFO] [dku.spark.exec]  - Loading exec env with impersonation enabled=false user=null
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.946] [main] [INFO] [dku.remoterun.fileexchange.helper]  - Running locally, dump error.json into cwd
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.955] [shutdown-hook-0] [INFO] [org.apache.spark.SparkContext]  - Invoking stop() from shutdown hook
[12:24:08] [INFO] [dku.utils]  - [2021/01/08-12:24:08.994] [shutdown-hook-0] [INFO] [org.spark_project.jetty.server.AbstractConnector]  - Stopped Spark@7c663eaf{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

 

0 Kudos
1 Solution
ismail
Level 2
Author
0 Kudos
1 Reply
ismail
Level 2
Author

Hi All,

Ignore this post..

Found the solution here,

https://doc.dataiku.com/dss/latest/spark/installation.html

 

0 Kudos