Build flow between two datasets

Recursive builds are a great feature. Build flow outputs reachable from here is a great feature. But in large projects, I also often find myself just wanting to constrain Dataiku to build a certain segment of my flow and nothing else. No upstream dependencies. No downstream outputs. Just start at one point, end at another, execute in order with all the dependency-calculation goodness of the Dataiku DAG.

Screenshot 2021-07-28 104417.jpg

For a simple example, I'd love to be able to select the leftmost and rightmost datasets here and with one click "run everything between" these datasets, resulting in my rightmost dataset being built, but nothing upstream of the leftmost dataset being rebuilt (since those take hours to finish and Dataiku's dependency management often triggers them to build even when nothing has changed). The dataset at the bottom wouldn't be rebuilt in this scenario.

For more general cases, it would also be really cool to get a preview of the sections of the flow that will be built every time, and have the option to add and remove items from that each time. That way as an alternative workflow, when selecting all of these and building or when building flow outputs reachable from here or when building recursively, I can just deselect the section I don't want rebuilt.

3 Comments
ktgross15
Dataiker

Thanks, this is in our backlog and I will note your interest - note that we also have this community product idea which is generally the same concept: https://community.dataiku.com/t5/Product-Ideas/Start-Flow-from-specific-dataset/idi-p/11809

Status changed to: In Backlog

Thanks, this is in our backlog and I will note your interest - note that we also have this community product idea which is generally the same concept: https://community.dataiku.com/t5/Product-Ideas/Start-Flow-from-specific-dataset/idi-p/11809

I'm wondering if doing this in a flow zone makes sense.  

So one could rebuild an entire flow zone as a way of selecting the scope of the re-build.

--Tom

I'm wondering if doing this in a flow zone makes sense.  

So one could rebuild an entire flow zone as a way of selecting the scope of the re-build.

MichaelG
Community Manager
Community Manager
 
I hope I helped! Do you Know that if I was Useful to you or Did something Outstanding you can Show your appreciation by giving me a KUDOS?

Looking for more resources to help you use DSS effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
Status changed to: In the Backlog