Number of spark executors

Options
ObadaJab
ObadaJab Registered Posts: 3 ✭✭✭

Hello,

I am using Spark-on-k8s to run my spark program. However, even though I am setting spark.executor.instances to 20(as indicated by the attached image), only 2 pods/executors are launched in the cluster. What could have gone wrong?

Capture.PNG

Answers

  • MehdiH
    MehdiH Dataiker, Dataiku DSS Core Designer, Dataiku DSS Core Concepts Posts: 21 Dataiker
    Options

    Hi @ObadaJab

    Thank you for reaching out.

    1. Can you check if there is no number of pods limitation set on your kubernetes cluster ?

    2. Do you have only one spark configuration? If not, make sure you are using the one where you set the spark.executor.instance to 20 for the job you are running

    3. Did you make sure to save the settings after modifying the number of instances ?

    We cannot say more about why it doesn't use more executors without logs, however.

    If you need further assistance for that, please open a support ticket and join a job diagnosis (Job page > Actions > Download diagnosis)

Setup Info
    Tags
      Help me…