Conditional execute of scenario step without steps failing or giving warnings

[Turribeach]

Hi,

I have a scenario that runs other scenarios as steps. One of these steps downloads some files from a third party website. In some instances there won't be any files downloaded so the next step fails as there are no files to process. I would like to be able to not execute all the remaining scenario steps when there are no files on my folder. However I don't want any steps to be set to warning or failing since it is possible that we won't get any files some days and that is a normal situation we don't want to trigger alerts for. 

There is a similar post already although this requires the use of metrics which will cause the step to fail or give a warning:

https://community.dataiku.com/t5/Using-Dataiku/Scenario-Conditional-on-Dataset-Metric/m-p/8474

Any ideas how to do this? I am thinking in using scenario variables but haven't seen a good example yet. Thanks

[fchataigner2]

Hi,

you can use the run condition on scenario steps for that. For example with a "Execute python code" like this one:

import dataiku
mf = dataiku.Folder('f')
contents = mf.list_paths_in_partition()

f_is_empty = len(contents) == 0

from dataiku.scenario import Scenario
Scenario().set_scenario_variables(f_is_empty=f_is_empty)

all the steps after this code can setup a run condition like:

Then when there are files in the folder "f", you get these subsequent steps run, and when there are no files, the steps are simply skipped

[Turribeach · Accepted solution]

Thanks fchataigner2, that's a valid solution that I initially thought of doing but then I decided to see if I could do it in a "clickers" way, without using Python code. With the help of Dataiku Support I came to this solution:

1. Create a scenario step to "Compute metrics" for the folder (let's call this step Compute_Metrics).
2. Next create a scenario step to "Define scenario variables"
3. On the Define scenario variables step toggle the "Evaluated variables" ON
4. Then define a new variable (let's call it number_of_files) with this formula: toNumber(filter(parseJson(stepOutput_Compute_Metrics)['ProjectID.FolderID_NP']['computed'], x, x["metricId"]=="basic:COUNT_FILES")[0].value)
5. You should replace ProjectID.FolderID with your corresponding values. Note that "Compute_Metrics" refers to the previous step name where you computed metrics for the folder
6. Finally in your conditional step set "Run this Step" to "If condition is satisfied" and the condition to: number_of_files >= 1

That's it! The step will conditionally execute based on the metric value of a folder. No failures or warnings.

[CoreyS]

Thank you for sharing your solution @Turribeach!

[Turribeach]

One last thing to add. By using the "If condition is satisfied" on a scenario step you can conditionally control if the step is executed based on the value of a metric, in this case the number of files on a folder. But using the "If condition is satisfied" option may also have an unwanted side effect. By default scenario steps only execute "if no prior step failed" so if you use the "If condition is satisfied" option to evaluate a variable you should take this  side effect into consideration. In my case I still wanted to make sure no steps executed if a prior step had failed, even those that use the "If condition is satisfied" option. So my actual condition ended up being this:

number_of_files >= 1 && outcome == 'SUCCESS'

As per the Step Flow Control documentation:

• outcome : the current outcome of the scenario; possible values are ‘SUCCESS’, ‘WARNING’, ‘FAILED’, ‘ABORTED’

So outcome holds the current state of the scenario which means that if any prior step failed outcome = FAILED and the conditional steps will not execute. Enjoy!