Passing Previous Month Partition to workflow recipes
In the scenario build, there is an option for selecting time partitions. This works great, but I want to know how to pass this variable to recipes such as a python and sql recipe. I wasn't able to make this work until I looked at the log and saw this variable '$DKU_DST_YEAR_MONTH', this appears to work, but I don't know if I should use this as I haven't found any documentation on how to accomplish this?
So I use it in my sql recipe like this
select * from mytable where partitioncol = '$DKU_DST_YEAR_MONTH'
What is the proper way to accomplish this?
Answers
-
Based on this article, I think this is a good way to accomplish this. It appears the scenario will pass the previous_month and that $DKU_DST_YEAR_MONTH variable becomes available. I wish Dataiku did a better job of making the variables visible.
Partitioning variables substitutions — Dataiku DSS 14 documentation