Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I have a custom plugin (recipe) in my project and I would like to access the recipe's configuration, specially the partition range:
In the code I have only access to these functions:
from dataiku.customrecipe import ( get_input_names_for_role, get_output_names_for_role, get_recipe_config, get_plugin_config, )
How can I get the values 2020-09-08 / 2020-09-10 ?
In partitioned code recipes, DSS exposes the partition identifiers of the input and output datasets via flow variables.
The available flow variables vary depending on the partition dependency function, partitioning dimensions, and the name and number of input/output dataset.
Therefore, the easiest way to find out which variables are available in a given code recipe, is to switch to the "variables" view on the recipe side-panel. Note that you may need to "validate" the code recipe for the variables to appear.
To insert the variables into your code, you can simply click on a variable name in the side-panel; this will insert the code dataiku.dku_flow_variables[<variable name>] into your script (in the case of python).
In the example shared above, I'm printing three of the available flow variables: the list of "Date" partitions from my source dataset, and the first and last "Date" partition of the source dataset. After running the recipe, the following is found in the logs:
Finally, note that in the case of a plugin recipe, you may need to programmatically generate the names of certain flow varaibles e.g. the variables that include the name of an input or output dataset.
Hopefully this helps!