How to automatically select the last partition of a partitioned file?
vincent_lr
Registered Posts: 1 ✭
Hello,
At each execution of a scenario, I historize my new data in a table parititioned by date of addition.
Then, from this partitioned file, what would be the best way to select the last available partition?
Thanks in advance,
Best Answer
-
Miguel Angel Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 118 Dataiker
Hi Vincent,
The most straightforward way to extract the latest partition from a dataset would be to use a recipe whose dependency function specifies ‘Latest available’.
This will give you the latest time partition among those already existing in the source dataset. Alternatively, alphabetical ordering is used for non-time-based dimensions.