The Dataiku Frontrunner Awards have just launched to recognize your achievements! Submit Your Entry

Accessing Spark web UI

ObadaJab
Level 1

Hello,

I am a beginner in Spark and I am trying to setup Spark on our Kubernetes cluster. The cluster is now working and I can run Spark jobs; however, I want to access Spark web UI to inspect how my job is being distributed. We usually port-forward a port(4040), but I am not being able to check which pod is the driver pod after running kubectl get pods --all-namespaces. What is the name DataIKU uses for Spark driver/master pod? 

TL;DR: how to access Spark web UI on a kubernetes-managed cluster?

Thanks in advance.

0 Replies