Jupyter Notebook integration with Google kubernetes engine

igour Registered Posts: 3
edited July 16 in Using Dataiku

We wanted to make GKE integration possible from notebook, my question is:

1) do we need to create and push the image first to the registry separately for notebook operations.

2) What all kind of execution is possible from notebook, example: python and pyspark? as I wanted to submit it to the GKE cluster directly from notebook.

3) Currently, when I tried to run the pyspark base code from notebook, it is throwing error, which means we have to set JAVA_HOME somewhere, would you please help me where to set this, do consider I am submitting load from notebook to GKE.

Error I am getting from pyspark code: I checked the pods logs on GKE it says JAVA_HOME is missing.

RuntimeError: Java gateway process exited before sending its port number

Please help.

Operating system used: rhel8


Setup Info
      Help me…