How to set a timeout for a particular Scenario build step via a custom Python step?

KimmyC

There is no explicit timeout functionality for a Build step within a DSS scenario. A common question is how to setup a timeout for a scenario or scenario step to avoid situations where a scenario gets stuck/hung in a running state indefinitely.

You can implement it using the Dataiku Python API. The same scenario step can be re-written as a custom Python step, in which case you can add additional Python code to implement a timeout.

Here is a code sample that you can try:

import time
import dataiku
from dataiku.scenario import Scenario, BuildFlowItemsStepDefHelper
from dataikuapi.dss.future import DSSFuture

s = Scenario()

# Define your build step below - this code is for specific building a dataset
step_handle = s.build_dataset("your_dataset", async=True)

start = time.time()
while not step_handle.is_done():
    end = time.time()
    print end, start, end-start
    # Define your timeout time - example below is for more than 1000 seconds
    if end - start > 1000:
        f = DSSFuture(dataiku.api_client(), step_handle.future_id)
        f.abort()
        raise 'Took too much time'