Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Get acces to partition target within Python Recipe

Solved!
s-cordo
Level 2
Get acces to partition target within Python Recipe

Hi, 

I would like to have access within my Python recipe to the partition that will be built. In fact, i would like to adapt my code considering the partiton targeted. 

The recipe takes as an input a non-partitioned hdfs dataset and as an output a file-based partitioned hdfs dataset which partition is categorical. 

I tried to use what is described there , but the function 

 

dataset.get_write_partition()

 

didn't work for me. 

Existing a way to accomplish what i want to ? 

Thanks 🙂 

PS : I'm using DSS 7.0 

dataiku_code_python_recipe.JPG

0 Kudos
1 Solution
dimitri
Dataiker
Dataiker

Hi @s-cordo 

You can access the partition name you want to build using the dku_flow_variables Python dictionary that you can access using dataiku.dku_flow_variables.
In your example, as your partitioning dimension name is thematique_name, you should be able to access its value using

dataiku.dku_flow_variables["DKU_DST_thematique_name"]

 

dataset.get_write_partition() is deprecated, we'll update the link you shared, thanks for the heads up!

Have a great day!

View solution in original post

0 Kudos
4 Replies
dimitri
Dataiker
Dataiker

Hi @s-cordo 

You can access the partition name you want to build using the dku_flow_variables Python dictionary that you can access using dataiku.dku_flow_variables.
In your example, as your partitioning dimension name is thematique_name, you should be able to access its value using

dataiku.dku_flow_variables["DKU_DST_thematique_name"]

 

dataset.get_write_partition() is deprecated, we'll update the link you shared, thanks for the heads up!

Have a great day!

View solution in original post

0 Kudos
s-cordo
Level 2
Author

Hi @dimitri , 

Thanks for your answer. 

However, I did not succeed to apply your solution.

dataiku_code_python_recipe_2.JPG²

Maybe

 

dataiku.dku_flow_variables["DKU_DST_thematique_name"]

doesn't exist for DSS 7.0  ?

I tried 

dataiku.get_flow_variables()

that looked equivalent, but i got a None value even if my partition seems to be well-defined on the flow : 

dataiku_code_python_recipe_3.JPG

Thanks for your help 

0 Kudos
dimitri
Dataiker
Dataiker

This is because you run the script from a notebook. Since the partition identifiers to build are configured on the recipe, they cannot be accessed from a notebook, and the dku_flow_variables dictionary is only defined when running from the recipe.

Note that both dataiku.dku_flow_variables and dataiku.get_flow_variables() will work and return the same result from the recipe, even with DSS 7.0.

Hope it helps!

0 Kudos
s-cordo
Level 2
Author

Indeed, it worked like a charm inside my python recipe 🙂 

Thanks a lot !

0 Kudos
A banner prompting to get Dataiku DSS