Submit your innovative use case or inspiring success story to the 2023 Dataiku Frontrunner Awards! LET'S GO

How to automatically select the last partition of a partitioned file?

Solved!
vincent_lr
Level 1
How to automatically select the last partition of a partitioned file?

Hello,


At each execution of a scenario, I historize my new data in a table parititioned by date of addition.
Then, from this partitioned file, what would be the best way to select the last available partition?


Thanks in advance,

0 Kudos
1 Solution
MiguelangelC
Dataiker

Hi Vincent,

The most straightforward way to extract the latest partition from a dataset would be to use a recipe whose dependency function specifies ‘Latest available’. 

This will give you the latest time partition among those already existing in the source dataset. Alternatively, alphabetical ordering is used for non-time-based dimensions.

View solution in original post

1 Reply
MiguelangelC
Dataiker

Hi Vincent,

The most straightforward way to extract the latest partition from a dataset would be to use a recipe whose dependency function specifies ‘Latest available’. 

This will give you the latest time partition among those already existing in the source dataset. Alternatively, alphabetical ordering is used for non-time-based dimensions.