Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
May I know if my error has got to do with the spark version?
Please advise what is the way forward? Thanks!
[12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.854] [main] [INFO] [org.apache.spark.SparkContext] - Created broadcast 0 from newAPIHadoopRDD at HDFSDatasetRDD.scala:160 [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.928] [main] [ERROR] [dku.spark.env] - job failed [12:24:08] [INFO] [dku.utils] - java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/DataFrame; [12:24:08] [INFO] [dku.utils] - at com.dataiku.dip.spark.StdDataikuSparkContext.getAsHadoopRDDDataFrameFromHDFS(StdDataikuSparkContext.scala:476) [12:24:08] [INFO] [dku.utils] - at com.dataiku.dip.spark.StdDataikuSparkContext.getAsSampledDataFrameInternal(StdDataikuSparkContext.scala:197) [12:24:08] [INFO] [dku.utils] - at com.dataiku.dip.spark.StdDataikuSparkContext.getAsSampledDataFrame(StdDataikuSparkContext.scala:145) [12:24:08] [INFO] [dku.utils] - at com.dataiku.dip.shaker.sparkimpl.ShakerSparkEntryPoint$.main(ShakerSparkEntryPoint.scala:37) [12:24:08] [INFO] [dku.utils] - at com.dataiku.dip.shaker.sparkimpl.ShakerSparkEntryPoint.main(ShakerSparkEntryPoint.scala) [12:24:08] [INFO] [dku.utils] - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [12:24:08] [INFO] [dku.utils] - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [12:24:08] [INFO] [dku.utils] - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [12:24:08] [INFO] [dku.utils] - at java.lang.reflect.Method.invoke(Method.java:498) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935) [12:24:08] [INFO] [dku.utils] - at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint] - Spark: initializeApplication [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint] - Spark: initializeApplication: ALREADY done [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.944] [main] [INFO] [dku.spark.entrypoint] - Loading exec env [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.945] [main] [INFO] [dku.spark.exec] - Loading exec env with impersonation enabled=false user=null [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.946] [main] [INFO] [dku.remoterun.fileexchange.helper] - Running locally, dump error.json into cwd [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.955] [shutdown-hook-0] [INFO] [org.apache.spark.SparkContext] - Invoking stop() from shutdown hook [12:24:08] [INFO] [dku.utils] - [2021/01/08-12:24:08.994] [shutdown-hook-0] [INFO] [org.spark_project.jetty.server.AbstractConnector] - Stopped Spark@7c663eaf{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
Hi All,
Ignore this post..
Found the solution here,
https://doc.dataiku.com/dss/latest/spark/installation.html
Hi All,
Ignore this post..
Found the solution here,
https://doc.dataiku.com/dss/latest/spark/installation.html