Queued Activities and Job Prioritization

somepunter
somepunter Registered Posts: 20 ✭✭✭

Is there a way to

  1. query the number of waiting activities due to insufficient slots
    i. at any given time
    ii. Historically?
  2. differentiate an activity in waiting status because it’s awaiting an upstream dependent activity vs a job slot vs global slot.
  3. prioritize jobs over others? E.g. a priority param per job?
  4. If not, can you share the scheduling algorithm? E.g. Is it round robin, FIFO per job or per activity?

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker

    Hi,

    1) No

    2) If it's waiting for a job slot, it will be in NOT_STARTED state. If it's waiting for a global slot, it will be in RUNNING state

    3) No

    4) Simple fair queue per limit (so at least, one per job, one global - plus additional one if you setup additional custom limits)

  • somepunter
    somepunter Registered Posts: 20 ✭✭✭

    Thanks,

    2a) in the case of awaiting global slot, how would I differentiate that against an actual running job?

    b) will the recorded start time be the time it actually runs or when it was ready to run?

    c) apologies, I can't find the enumeration of all statuses in your docs, could you point me to it please?

Setup Info
    Tags
      Help me…