Survey banner
Switching to Dataiku - a new area to help users who are transitioning from other tools and diving into Dataiku! CHECK IT OUT

Number of spark executors

Level 1
Number of spark executors


I am using Spark-on-k8s to run my spark program. However, even though I am setting spark.executor.instances to 20(as indicated by the attached image), only 2 pods/executors are launched in the cluster. What could have gone wrong?


0 Kudos
1 Reply

Hi @ObadaJab 

Thank you for reaching out.

1. Can you check if there is no number of pods limitation set on your kubernetes cluster ?

2. Do you have only one spark configuration? If not, make sure you are using the one where you set the spark.executor.instance to 20 for the job you are running

3. Did you make sure to save the settings after modifying the number of instances ?  

We cannot say more about why it doesn't use more executors without logs, however.

If you need further assistance for that, please open a support ticket and join a job diagnosis (Job page > Actions > Download diagnosis)

0 Kudos