Discover the winners & finalists of the 2022 Dataiku Frontrunner Awards!READ THEIR USE CASES

Accessing Spark web UI

Solved!
ObadaJab
Level 1
Accessing Spark web UI

Hello,

I am a beginner in Spark and I am trying to setup Spark on our Kubernetes cluster. The cluster is now working and I can run Spark jobs; however, I want to access Spark web UI to inspect how my job is being distributed. We usually port-forward a port(4040), but I am not being able to check which pod is the driver pod after running kubectl get pods --all-namespaces. What is the name DataIKU uses for Spark driver/master pod? 

TL;DR: how to access Spark web UI on a kubernetes-managed cluster?

Thanks in advance.

1 Solution
Omar
Dataiker

Hi,

the Spark driver is always executed in the DSS box, not in k8s.

You should look for the process using port 4040 on the DSS box (or increments of that, if they are taken already).

Also consider that spark drivers are specific for the job you're executing and not persistent: once the job is completed, they are destroyed. Another job will start its own driver.

 

Take care,

 

Omar 
FieldEngineer @ Dataiku

View solution in original post

0 Kudos
1 Reply
Omar
Dataiker

Hi,

the Spark driver is always executed in the DSS box, not in k8s.

You should look for the process using port 4040 on the DSS box (or increments of that, if they are taken already).

Also consider that spark drivers are specific for the job you're executing and not persistent: once the job is completed, they are destroyed. Another job will start its own driver.

 

Take care,

 

Omar 
FieldEngineer @ Dataiku

0 Kudos