Long running job - where to find in logs?
Hi all I was wondering where I can find the information display in the UI about long running jobs?
In the UI this section is in administration -> monitoring -> running background tasks
where can I find this information in a data directory?
Additionally, is there a way to send a notification when a job is still running after certain amount of time?
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,032 Neuron
@cyber
wrote:Hi all I was wondering where I can find the information display in the UI about long running jobs?
In the UI this section is in administration -> monitoring -> running background tasks
where can I find this information in a data directory?
The data directly has the logs for running jobs but no easy way to determine which ones are running. For that you should use the Dataiku Python API which at the moment doesn't have an API Call that can be used for all running jobs in an instance. See this product idea and vote for it.
Additionally, is there a way to send a notification when a job is still running after certain amount of time?
Because of this limitation with the Python API then it's hard to monitor all the instance jobs as you have to query project by project which in a large instance is not practical or even feasible. Here is a sample of that approach. Another approach is to build your scenarios steps with a timeout option. Here is a sample of that approach. That could also be changed to send an email when the step exceeds a set threshold.
So as you can see we are sadly very limited on how we can monitor running jobs in Dataiku until Dataiku implements a running jobs API.