Problems using async in scenarios

John_Kelly
John_Kelly Registered Posts: 16 ✭✭✭✭
edited July 18 in Using Dataiku

I've gotten this error in many scenarios across a lot of versions of DSS, and always assumed it surely must be a known bug and that it would be fixed. But by this point I'm starting to wonder if somehow I'm doing something wrong. Using async = True seems pretty straight forward, but a lot of times I end up with the error below. Any ideas?

Thanks,

John




[2018/12/16-17:15:27.415] [Exec-279] [INFO] [process] - Traceback (most recent call last):
[2018/12/16-17:15:27.415] [Exec-279] [INFO] [process] - File "/home/dataiku/dss_data/scenarios/ERM_MASTER_PORTFOLIO/BUILD_MODEL_OUTPUTS/2018-12-16-17-10-38-992/custom-scenario/script.py", line 27, in <module>
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - while not sev_br.is_done():
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/scenario/step.py", line 98, in is_done
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - }, err_msg="Failed to track step future")
[2018/12/16-17:15:27.416] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/core/intercom.py", line 82, in backend_json_call
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - return _handle_json_resp(backend_api_post_call(path, data, **kwargs), err_msg = err_msg)
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - File "/home/dataiku/dataiku-dss-5.0.4/python/dataiku/core/intercom.py", line 148, in _handle_json_resp
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - raise Exception("%s: %s" % (err_msg, _get_error_message(err_data).encode("utf8")))
[2018/12/16-17:15:27.417] [Exec-279] [INFO] [process] - Exception: Failed to track step future: JobID not found : OwNLUtij. Running jobs: ["LZ7ni3cz","KHyjzJ2a","GtxHfmlh","1h3oQSvT","cd9JhCAF","JViR3tLW","z5gJUDu5"]

Best Answer

  • Alex_Combessie
    Alex_Combessie Alpha Tester, Dataiker Alumni Posts: 539 ✭✭✭✭✭✭✭✭✭
    edited July 18 Answer ✓

    Hi John,

    We had a deeper look at this issue. The explanation is that after .is_done() has returned True once, it cannot be called again. So ideally you should write something like


    scenario_1_done = False
    scenario_2_done = False

    while not scenario_1_done or not scenario_2_done:
    if not scenario_1_done:
    scenario_1_done = scenario_1.is_done()
    if not scenario_2_done:
    scenario_2_done = scenario_2.is_done()
    time.sleep(1)

    Hope it helps,

    Alex

Answers

  • John_Kelly
    John_Kelly Registered Posts: 16 ✭✭✭✭
    Thanks, this appears to be working! This also explains why when I did it the way suggested in the example code:

    while not scenario_1.is_done() or not scenario_2.is_done():
    time.sleep(1)

    that it seemed to always fail, while using nested loops the way that I did only failed sometimes. With nested loops, it should only fail if the scenario being checked by the inner loop finishes first.
Setup Info
    Tags
      Help me…