dataset.build() doesn't work for streaming python recipe output dataset

Options
Sangavi_M
Sangavi_M Registered Posts: 6

I was trying to start a job in order to build the dataset, whose recipe parent is a very simple streaming python recipie. It doesn't throw any error but doesnt build the dataset as well

But same works fine with a normal python recipe and I'm able to build a dataset.

is there any way i can achieve the same for a streaming python recipie? This is what I was doing:

client = dataiku.api_client()
project = client.get_default_project()


sample_df = project.get_dataset("sample")
job = sample_df.build()
print(job)

Best Answer

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,725 Neuron
    edited July 17 Answer ✓
    Options

    Continuous recipes are a different beast and you need to use special methods to start and stop them:

    import dataiku
    
    client = dataiku.api_client()
    project = client.get_project('some project key with continuous activities')
    continuous_activities = project.list_continuous_activities()
    
    for recipe in continuous_activities:
        recipe_running = str(recipe.get_status()['mainLoopState']['futureInfo']['alive'])
        print(str(recipe.recipe_id) + " - Running: " + recipe_running)
        if recipe_running == "False":
            recipe.start()

Answers

Setup Info
    Tags
      Help me…