Number of spark executors

ObadaJab
ObadaJab Registered Posts: 3 ✭✭✭
in Setup & Configuration

Hello,

I am using Spark-on-k8s to run my spark program. However, even though I am setting spark.executor.instances to 20(as indicated by the attached image), only 2 pods/executors are launched in the cluster. What could have gone wrong?

Capture.PNG

· Share on FacebookShare on Twitter

Answers

  • MehdiH
    MehdiH Dataiker, Dataiku DSS Core Designer, Dataiku DSS Core Concepts Posts: 21 Dataiker

    Hi @ObadaJab

    Thank you for reaching out.

    1. Can you check if there is no number of pods limitation set on your kubernetes cluster ?

    2. Do you have only one spark configuration? If not, make sure you are using the one where you set the spark.executor.instance to 20 for the job you are running

    3. Did you make sure to save the settings after modifying the number of instances ?

    We cannot say more about why it doesn't use more executors without logs, however.

    If you need further assistance for that, please open a support ticket and join a job diagnosis (Job page > Actions > Download diagnosis)

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…