How to automatically select the last partition of a partitioned file?

Options
vincent_lr
vincent_lr Registered Posts: 1

Hello,


At each execution of a scenario, I historize my new data in a table parititioned by date of addition.
Then, from this partitioned file, what would be the best way to select the last available partition?


Thanks in advance,

Tagged:

Best Answer

  • Miguel Angel
    Miguel Angel Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 118 Dataiker
    Answer ✓
    Options

    Hi Vincent,

    The most straightforward way to extract the latest partition from a dataset would be to use a recipe whose dependency function specifies ‘Latest available’.

    This will give you the latest time partition among those already existing in the source dataset. Alternatively, alphabetical ordering is used for non-time-based dimensions.

Setup Info
    Tags
      Help me…