The India User Group is live! Be a part of our first Indian user event: JOIN THE EVENT

Recipes with Time Range dependence partitions: is it possible to ignore missing partitions?

Solved!
Ignacio_Toledo
Recipes with Time Range dependence partitions: is it possible to ignore missing partitions?

While working with a team partner, we started to see an unexpected behavior with a join recipe.

We are trying to join 2 partitioned datasets, both at the day level, and we are using this dependence definition:

Selection_013.png

 So we want to join the first dataset with 14 partitions from the second dataset at a time. The problem happens when we want to process "today" (or for that matter, any of the last 7 days) and the recipe will fail because it can't not find the partitions "today+1", "today+2", .. "today+7".

Is there any way to setup the time range dependence to ignore this error so the data can still be joined, instead of failing completely? We think this should be an option (ignore missing partition) because we have several other use cases where the same happens: if in the time range one partition is missing, the whole processing day fails.

The solution "do not add days into the future" is not valid for us because some times we need to reprocess all the data in the last 2 weeks or so.

Thanks!

0 Kudos
1 Solution
CoreyS
Community Manager
Community Manager

Hi @Ignacio_Toledo thank you for your question and I think you've provided pretty good detail and context.

Short answer to your original question is that it is not possible to ignore missing partitions but I am working on getting you a more detailed response. Thanks again for your question!

Looking for more resources to help you use DSS effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as ‘Accepted Solution’ to help others like you!

View solution in original post

3 Replies
Ignacio_Toledo
Author

@CoreyS or any other community manager: do you think that this post is missing information to make it understandable or should I posted in another thread?

If the issue is understandable and I get the confirmation that what we need can't be done currently, they I think I'll suggest it as a new idea in the Community Feedback section.

Thanks!

0 Kudos
CoreyS
Community Manager
Community Manager

Hi @Ignacio_Toledo thank you for your question and I think you've provided pretty good detail and context.

Short answer to your original question is that it is not possible to ignore missing partitions but I am working on getting you a more detailed response. Thanks again for your question!

Looking for more resources to help you use DSS effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as ‘Accepted Solution’ to help others like you!

View solution in original post

Ignacio_Toledo
Author

Thanks @CoreyS! I'll wait for your more detailed response before creating a Suggestion in the community feedback section.

Cheers!

0 Kudos
Labels (2)
A banner prompting to get Dataiku DSS
Public