Submit your use case or success story to the 2023 edition of the Dataiku Frontrunner Awards ENTER YOUR SUBMISSION

Queued Activities and Job Prioritization

somepunter
Level 3
Queued Activities and Job Prioritization

Is there a way to

  1. query the number of waiting activities due to insufficient slots
    i. at any given time
    ii. Historically?
  2. differentiate an activity in waiting status because it’s awaiting an upstream dependent activity vs a job slot vs global slot.
  3. prioritize jobs over others? E.g. a priority param per job?
  4. If not, can you share the scheduling algorithm? E.g. Is it round robin, FIFO per job or per activity?
0 Kudos
2 Replies
Clément_Stenac
Dataiker

Hi,

1) No

2) If it's waiting for a job slot, it will be in NOT_STARTED state. If it's waiting for a global slot, it will be in RUNNING state

3) No

4) Simple fair queue per limit (so at least, one per job, one global - plus additional one if you setup additional custom limits)

0 Kudos
somepunter
Level 3
Author

Thanks,

2a) in the case of awaiting global slot, how would I differentiate that against an actual running job?

b) will the recorded start time be the time it actually runs or when it was ready to run?

c) apologies, I can't find the enumeration of all statuses in your docs, could you point me to it please?

0 Kudos

Labels

?

Setup info

?
A banner prompting to get Dataiku