Queued Activities and Job Prioritization

Registered Posts: 20 ✭✭✭

Is there a way to

  1. query the number of waiting activities due to insufficient slots
    i. at any given time
    ii. Historically?
  2. differentiate an activity in waiting status because it’s awaiting an upstream dependent activity vs a job slot vs global slot.
  3. prioritize jobs over others? E.g. a priority param per job?
  4. If not, can you share the scheduling algorithm? E.g. Is it round robin, FIFO per job or per activity?

Answers

  • Dataiker, Dataiku DSS Core Designer, Registered, Moderator Posts: 753 Dataiker

    Hi,

    1) No

    2) If it's waiting for a job slot, it will be in NOT_STARTED state. If it's waiting for a global slot, it will be in RUNNING state

    3) No

    4) Simple fair queue per limit (so at least, one per job, one global - plus additional one if you setup additional custom limits)

  • Registered Posts: 20 ✭✭✭

    Thanks,

    2a) in the case of awaiting global slot, how would I differentiate that against an actual running job?

    b) will the recorded start time be the time it actually runs or when it was ready to run?

    c) apologies, I can't find the enumeration of all statuses in your docs, could you point me to it please?

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.