I would like to have access within my Python recipe to the partition that will be built. In fact, i would like to adapt my code considering the partiton targeted.
The recipe takes as an input a non-partitioned hdfs dataset and as an output a file-based partitioned hdfs dataset which partition is categorical.
I tried to use what is described there , but the function
didn't work for me.
Existing a way to accomplish what i want to ?
PS : I'm using DSS 7.0
You can access the partition name you want to build using the dku_flow_variables Python dictionary that you can access using dataiku.dku_flow_variables.
In your example, as your partitioning dimension name is thematique_name, you should be able to access its value using
dataset.get_write_partition() is deprecated, we'll update the link you shared, thanks for the heads up!
Have a great day!
Hi @dimitri ,
Thanks for your answer.
However, I did not succeed to apply your solution.
doesn't exist for DSS 7.0 ?
that looked equivalent, but i got a None value even if my partition seems to be well-defined on the flow :
Thanks for your help
This is because you run the script from a notebook. Since the partition identifiers to build are configured on the recipe, they cannot be accessed from a notebook, and the dku_flow_variables dictionary is only defined when running from the recipe.
Note that both dataiku.dku_flow_variables and dataiku.get_flow_variables() will work and return the same result from the recipe, even with DSS 7.0.
Hope it helps!