Recipes with Time Range dependence partitions: is it possible to ignore missing partitions?

Ignacio_Toledo

While working with a team partner, we started to see an unexpected behavior with a join recipe.

We are trying to join 2 partitioned datasets, both at the day level, and we are using this dependence definition:

 So we want to join the first dataset with 14 partitions from the second dataset at a time. The problem happens when we want to process "today" (or for that matter, any of the last 7 days) and the recipe will fail because it can't not find the partitions "today+1", "today+2", .. "today+7".

Is there any way to setup the time range dependence to ignore this error so the data can still be joined, instead of failing completely? We think this should be an option (ignore missing partition) because we have several other use cases where the same happens: if in the time range one partition is missing, the whole processing day fails.

The solution "do not add days into the future" is not valid for us because some times we need to reprocess all the data in the last 2 weeks or so.

Thanks!

CoreyS

Hi @Ignacio_Toledo
thank you for your question and I think you've provided pretty good detail and context.

Short answer to your original question is that it is not possible to ignore missing partitions but I am working on getting you a more detailed response. Thanks again for your question!

Ignacio_Toledo

@CoreyS
or any other community manager: do you think that this post is missing information to make it understandable or should I posted in another thread?

If the issue is understandable and I get the confirmation that what we need can't be done currently, they I think I'll suggest it as a new idea in the Community Feedback section.

Thanks!

Ignacio_Toledo

Thanks @CoreyS
! I'll wait for your more detailed response before creating a Suggestion in the community feedback section.

Cheers!