The Dataiku Frontrunner Awards have just launched to recognize your achievements! Submit Your Entry

Accessing Spark web UI

Level 1


I am a beginner in Spark and I am trying to setup Spark on our Kubernetes cluster. The cluster is now working and I can run Spark jobs; however, I want to access Spark web UI to inspect how my job is being distributed. We usually port-forward a port(4040), but I am not being able to check which pod is the driver pod after running kubectl get pods --all-namespaces. What is the name DataIKU uses for Spark driver/master pod? 

TL;DR: how to access Spark web UI on a kubernetes-managed cluster?

Thanks in advance.

0 Replies