Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
The documentation seems quite sparse on this, although if there is a good one stop document for this I would appreciate it.
I am trying to understand the right way to:
Any help on this would be appreciated, even if links to relevant docs.
At the moment the code I have for accessing the output of the SQL step is below.
from dataiku.scenario import Scenario import dataiku s = Scenario() print('Python Step Ran...ETL date must be today..') outputs = s.get_previous_steps_outputs() sql_output = [o['result'] for o in outputs if o["stepName"] == 'CheckSapphireETLDate'] s.set_scenario_variables(sapphire_etl_date = sql_output['rows']) print(f'sapphire_etl_date = ' + dataiku.get_custom_variables()["sapphire_etl_date"])
I thought I'd share my experiences as they related to your questions. Not complete answers but maybe still helpful.
I believe I explored programmatically setting the outcome of the step using Python but wasn't able to figure out a solution. Well, one can simply raise a Python exception to indicate a failure (and that's what I do when I want to fail the step) but that's a bit different than setting the result variable directly.
I ended up using a different variable that I created and set to defined value in the first step in the scenario. Not ideal but it worked to accomplish what I was trying to do which was track multiple outcomes and then generate a different email for each outcome.
The scenario get_all_variables() method is used to get both project and scenario variables within a Python step.
To get the output of a SQL script in Python, I would just execute it within the Python step rather than as a separate step. See documentation here: https://doc.dataiku.com/dss/latest/python-api/sql.html
Here's an example of getting SQL step output using the formula language: parseJson(stepOutput_TimelinessCheck)['rows']. In this example, "TimelinessCheck" is the label given to the step.
I don't know how you would set the output of a python step besides via a variable. It maybe possible, I just don't know.
Hope this helps.
Thanks. Very helpful. I think I have the pattern down now for setting variables and using them in expressions as conditions in future steps. I assume though that it must be possible to set the outcome of the step via Python to success or failure, rather than creating your own variables and using them in expressions.