Problems using async in scenarios
![John_Kelly](https://us.v-cdn.net/6038231/uploads/Dataiku/nAvatar16.png)
John_Kelly
Registered Posts: 16 ✭✭✭✭
I've gotten this error in many scenarios across a lot of versions of DSS, and always assumed it surely must be a known bug and that it would be fixed. But by this point I'm starting to wonder if somehow I'm doing something wrong. Using async = True seems pretty straight forward, but a lot of times I end up with the error below. Any ideas?
Thanks,
John
[2018/12/16-17:15:27.415] [Exec-279] [INFO] [process] - Traceback (most recent call last):
[2018/12/16-17:15:27.415] [Exec-279] [INFO] [process] - File "/home/dataiku/dss_data/scenarios/ERM_MASTER_PORTFOLIO/BUILD_MODEL_OUTPUTS/2018-12-16-17-10-38-992/custom-scenario/script.py", line 27, in <module>
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - while not sev_br.is_done():
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/scenario/step.py", line 98, in is_done
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - }, err_msg="Failed to track step future")
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/core/intercom.py", line 82, in backend_json_call
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - return _handle_json_resp(backend_api_post_call(path, data, **kwargs), err_msg = err_msg)
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/core/intercom.py", line 148, in _handle_json_resp
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - raise Exception("%s: %s" % (err_msg, _get_error_message(err_data).encode("utf8")))
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - Exception: Failed to track step future: JobID not found : OwNLUtij. Running jobs: ["LZ7ni3cz","KHyjzJ2a","GtxHfmlh","1h3oQSvT","cd9JhCAF","JViR3tLW","z5gJUDu5"]
Best Answer
-
Hi John,
We had a deeper look at this issue. The explanation is that after .is_done() has returned True once, it cannot be called again. So ideally you should write something like
scenario_1_done = False
scenario_2_done = False
while not scenario_1_done or not scenario_2_done:
if not scenario_1_done:
scenario_1_done = scenario_1.is_done()
if not scenario_2_done:
scenario_2_done = scenario_2.is_done()
time.sleep(1)Hope it helps,
Alex
Answers
-
Thanks, this appears to be working! This also explains why when I did it the way suggested in the example code:
while not scenario_1.is_done() or not scenario_2.is_done():
time.sleep(1)
that it seemed to always fail, while using nested loops the way that I did only failed sometimes. With nested loops, it should only fail if the scenario being checked by the inner loop finishes first.