Pyspark notebooks with custom container image

Options
DrissiReda
DrissiReda Registered Posts: 57 ✭✭✭✭✭

Hello,

I am trying to make a custom dataiku container image to run pyspark on notebooks.

I'm running inside a kubernetes cluster, with no docker installed, so I can't let dss build my images.

I already have working images of spark and containers. But when I add spark and java to the container image I have two problems:

1. pyspark isn't included in the pythonpath and the overridden variable in the dockerfile is ignored inside the runner.

2. After I add it manually I have an error:

Exception in thread "main" org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: nameat com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)at javax.security.auth.login.LoginContext.login(LoginContext.java:587)at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)at scala.Option.getOrElse(Option.scala:189)at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)at scala.Option.map(Option.scala:230)at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)at scala.Option.getOrElse(Option.scala:189)at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)at scala.Option.map(Option.scala:230)at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: nameat com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Method.invoke(Method.java:498)at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)at javax.security.auth.login.LoginContext.login(LoginContext.java:587)at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)at scala.Option.getOrElse(Option.scala:189)at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412)at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:79)at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.secMgr$1(SparkSubmit.scala:368)at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:376)at scala.Option.map(Option.scala:230)at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:376)at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)at javax.security.auth.login.LoginContext.login(LoginContext.java:587)at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)... 19 moreTraceback (most recent call last):File "<stdin>", line 1, in <module>File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 376, in getOrCreateFile "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 133, in __init__File "/opt/spark/python/lib/pyspark.zip/pyspark/context.py", line 325, in _ensure_initializedFile "/opt/spark/python/lib/pyspark.zip/pyspark/java_gateway.py", line 105, in launch_gatewayException: Java gateway process exited before sending its port number

Is there a better way to build the container image dockerfile? How can I fix this error?

Pyspark recipes work well, but they use the spark image (used for spark-submit), is there a way to make the notebook kernels use the spark image instead of the container image?

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer Posts: 753 Dataiker
    Options

    Hi,

    Unfortunately, for the image build, you're extremely far from a regular setup, and we cannot really provide support with building images in such an exotic setup.

    As for your second question, to run Pyspark notebooks, the notebook itself must not run in containers. i.e. the notebook must run with "Local execution". If your default Spark config is configured to use Kubernetes, then the executors will be on Kubernetes as expected.

  • DrissiReda
    DrissiReda Registered Posts: 57 ✭✭✭✭✭
    Options

    Thanks, that actually works well for me, but I'm getting this error:

    Py4JJavaError: An error occurred while calling o51.count.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 7, 10.244.235.146, executor 316): java.net.UnknownHostException: dataiku-0

    How can I change the host from "dataiku-0" to "dataiku-0.dataiku" . Also I don't understand why the pyspark shell is trying to contact my dss pod.

Setup Info
    Tags
      Help me…