Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Preempt the job queue

I often run into this message when building datasets by hand:

 

[2022/05/13-11:08:07.403] [ActivityExecutor-32] [INFO] [dku.flow.jobrunner] running compute_allocated_liners_NP - Waiting for a free slot before executing this activity...
[2022/05/13-11:08:07.410] [ActivityExecutor-32] [INFO] [dku.flow.jobrunner] running compute_allocated_liners_NP - Reason: The global limit of 12 running activities has been reached.

 

 

Without lifting the global limit, I'd like to be able to put the job I'm working on at the top of the queue, preempting whatever's already running, sending it back to the queue to be restarted after the job I selected is completed. This will speed up the workflow in big projects that have a lot of data moving all the time.