How to set a timeout for a particular Scenario build step via a custom Python step?

KimmyC Dataiker Posts: 34 Dataiker

There is no explicit timeout functionality for a Build step within a DSS scenario. A common question is how to setup a timeout for a scenario or scenario step to avoid situations where a scenario gets stuck/hung in a running state indefinitely.

You can implement it using the Dataiku Python API. The same scenario step can be re-written as a custom Python step, in which case you can add additional Python code to implement a timeout.

Here is a code sample that you can try:

import timeimport dataikufrom dataiku.scenario import Scenario, BuildFlowItemsStepDefHelperfrom dataikuapi.dss.future import DSSFutures = Scenario()# Define your build step below - this code is for specific building a datasetstep_handle = s.build_dataset("your_dataset", async=True)start = time.time()while not step_handle.is_done():end = time.time()print end, start, end-start# Define your timeout time - example below is for more than 1000 secondsif end - start > 1000:f = DSSFuture(dataiku.api_client(), step_handle.future_id)f.abort()raise 'Took too much time'
Setup Info
      Help me…