Recipes with Time Range dependence partitions: is it possible to ignore missing partitions?

Options
Ignacio_Toledo
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron

While working with a team partner, we started to see an unexpected behavior with a join recipe.

We are trying to join 2 partitioned datasets, both at the day level, and we are using this dependence definition:

Selection_013.png

 So we want to join the first dataset with 14 partitions from the second dataset at a time. The problem happens when we want to process "today" (or for that matter, any of the last 7 days) and the recipe will fail because it can't not find the partitions "today+1", "today+2", .. "today+7".

Is there any way to setup the time range dependence to ignore this error so the data can still be joined, instead of failing completely? We think this should be an option (ignore missing partition) because we have several other use cases where the same happens: if in the time range one partition is missing, the whole processing day fails.

The solution "do not add days into the future" is not valid for us because some times we need to reprocess all the data in the last 2 weeks or so.

Thanks!

Best Answer

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭
    Answer ✓
    Options

    Hi @Ignacio_Toledo
    thank you for your question and I think you've provided pretty good detail and context.

    Short answer to your original question is that it is not possible to ignore missing partitions but I am working on getting you a more detailed response. Thanks again for your question!

Answers

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    @CoreyS
    or any other community manager: do you think that this post is missing information to make it understandable or should I posted in another thread?

    If the issue is understandable and I get the confirmation that what we need can't be done currently, they I think I'll suggest it as a new idea in the Community Feedback section.

    Thanks!

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Thanks @CoreyS
    ! I'll wait for your more detailed response before creating a Suggestion in the community feedback section.

    Cheers!

Setup Info
    Tags
      Help me…