New to Dataiku DSS? Try out our NEW Quick Start Programs today and get onboarded on the product in just one hour! Let's go

Accessing Spark web UI

ObadaJab
Level 1
Accessing Spark web UI

Hello,

I am a beginner in Spark and I am trying to setup Spark on our Kubernetes cluster. The cluster is now working and I can run Spark jobs; however, I want to access Spark web UI to inspect how my job is being distributed. We usually port-forward a port(4040), but I am not being able to check which pod is the driver pod after running kubectl get pods --all-namespaces. What is the name DataIKU uses for Spark driver/master pod? 

TL;DR: how to access Spark web UI on a kubernetes-managed cluster?

Thanks in advance.

0 Replies