Usage of dku_flow_variables

SanderVW

Part of our ETL is using dku_flow_variables to determine the period we are processing data for. I wanted to see what this does in the dataiku api documentation (Python - Dataiku Developer Guide) but could not find it there. Can anyone explain the usage of dku_flow_variables or point me to documentation of this?

The way we use it currently is in Python using the code below but I don't know how Dataiku determines these variables as the partition we have is a MONTH partitioning with a string like this: "2025-03"

year = dataiku.dku_flow_variables["DKU_DST_YEAR"]

month = int(dataiku.dku_flow_variables["DKU_DST_MONTH"])

What I am interested in specifically is when these variables update and how to see all variables available.

Operating system used: Windows

ThierryC

You have a definition of partitioning variables in this documentation : https://doc.dataiku.com/dss/latest/partitions/variables.html

You will find a table to explain DKU_DST_YEAR or DKU_DST_MONTH.

Let us know if it helps or if you need something different.

Turribeach

Can you actually post your requirement? What are you trying to achieve?

SanderVW

https://community.dataiku.com/discussion/comment/46035#Comment_46035

I am not the person who set up the code originally but would like to know how it operates so we can use it for similar cases. I would like to get an overview of all the flow variables there are since I would not know how to find out about the DKU_DST_YEAR and DKU_DST_MONTH variables otherwise.

Our current implementation uses the code in my post to determine the month partition in a Python script so that we can manipulate our data in the script based on the specific month. If this is also something that we could do in prepare recipes, for instance, it would be good to know.

If it's not quite considered best practice that's also good to know but I'd still like to know how it works either way. Is it documented anywhere?

SanderVW

https://community.dataiku.com/discussion/comment/46048#Comment_46048

Thank you very much, this is the overview I was looking for! 😊

Am I correct in saying you can only use these variables from code recipes? Meaning there's no way to refer to them within visual recipes like prepare or join steps?