Meet DSS user Ben Powis, Data Science Manager at UK retail company MandM Direct Read More

Custom recipe from plugin - read recipe configuration

Level 4
Custom recipe from plugin - read recipe configuration

Hi, 

I have a custom plugin (recipe) in my project and I would like to access the recipe's configuration, specially the partition range:

image.png

 

In the code I have only access to these functions:

 

 

from dataiku.customrecipe import (
get_input_names_for_role,
get_output_names_for_role,
get_recipe_config,
get_plugin_config,
)

 

 

How can I get the values 2020-09-08 / 2020-09-10 ?

 

0 Kudos
1 Reply
Dataiker
Dataiker

Hi Tomas,

In partitioned code recipes, DSS exposes the partition identifiers of the input and output datasets via flow variables.

The available flow variables vary depending on the partition dependency function, partitioning dimensions, and the name and number of input/output dataset.

Therefore, the easiest way to find out which variables are available in a given code recipe, is to switch to the "variables" view on the recipe side-panel. Note that you may need to "validate" the code recipe for the variables to appear.

p1.png

To insert the variables into your code, you can simply click on a variable name in the side-panel; this will insert the code dataiku.dku_flow_variables[<variable name>] into your script (in the case of python).

In the example shared above, I'm printing three of the available flow variables: the list of "Date" partitions from my source dataset, and the first and last "Date" partition of the source dataset. After running the recipe, the following is found in the logs: 

p2.png

Finally, note that in the case of a plugin recipe, you may need to programmatically generate the names of certain flow varaibles e.g. the variables that include the name of an input or output dataset.

Hopefully this helps!

Ned

0 Kudos