Want to Stop Rebuilding "Expensive" Parts of your Flow? Explicit Builds are the Answer!READ MORE

Move objects and Zones on the Flow

I think it would be very useful to be able to move objects and flow zones around in the flow display. It appears Dataiku determines where each recipe, dataset, etc go in the flow and I cannot edit that. I have used Alteryx in the past and it had that ability, which I liked. It allows me to organize the flow however I see fit and make it more visually appealing and more understandable for someone looking at it.

6 Comments
ktgross15
Dataiker
Dataiker

Hi @Ben_Rutan ,

Thanks for the feedback 🙂

The solution to managing large flows is by using flow zones. Through flow zones, you can split your flow into small, named/colored zones, which makes it much easier to orient yourself and focus only where you need to, especially when working with a very large flow. 

Have you worked with flow zones/found them helpful in managing large flows? Happy to collect any other feedback here, and I will log your request, as I know that flow zones wasn't exactly what you were looking for, but is a solution that many users have found helpful 🙂

Katie

CoreyS
Community Manager
Community Manager
Status changed to: In Backlog
 
Ben_Rutan
Level 2

Hi @ktgross15 ,

I have used flow zones and they are somewhat helpful, but I think it would be more beneficial if I could rearrange each recipe, dataset, etc however I see fit instead of how Dataiku decides to display them. Just something to add to the wish list 🙂

Ben

Turribeach
Level 6

In my experience Dataiku does a great job it drawing the flow in a reasonable way. There are indeed cases where things do look a bit awkward (see screen shot below) but even in this situation I don't think I will spend the time it would take to reorg the flow. And any manual flow reorg would be subject to manual maintenance as well, meaning new flow objects will need to be placed in sensible places so I don't think a lot of people will be willing to invest their time on this. Just look at the mess that are most people's Windows desktop or the iPhone App home page.

Screenshot 2022-07-06 at 7.56.04 pm.png

importthepandas
Level 4

Agreed! Moving and nesting flow zones - organizing components more effectively in sub sections in a flow zone.

AshleyW
Dataiker
Dataiker
Status changed to: Not Planned

Hi @Ben_Rutan ,

Thanks for this suggestion! This idea it not currently planned, but we’ll circle back to this thread if we’re looking for more info in the future.

The Flow is the visual representation of how datasets, recipes, and models work together to move data through an analytical pipeline. Projects can quickly become complex with hundreds of objects on the Flow. We sometimes call these ‘spaghetti’ flows 🙂

As it grows in complexity, manually positioning objects and zones contributes to an unnecessary burden to maintain this arrangement over time, as @Turribeach  mentioned. We’ve also noticed that manual positioning is very personal: while it may facilitate one person’s understanding of a project, it often does the opposite for others.

To help with organizing objects in a complex project, I’ll second @ktgross15 ’s suggestion to use zones. Since Dataiku projects are built around the notion of collaboration, we find that a layout structured around flow zones is more neutral than moving things around.

At Dataiku, we use zones with great success to arrange datasets/recipes/models into logical groupings, making large flows much easier to manage and collaborate on. They’re automatically laid out for me, and I never worry about them colliding or needing to resize them. With a description in the R pane, it’s easy for me (and others) to understand what a zone contributes to the project and see how all of the pieces of the pipeline fit together. (Yes, it drives me nuts when objects move; I've found zones help with that too)

As a bonus, check out the ‘Flow Zones’ flow view of a project. Clicking the ‘hide zones’ button can be a surprising reminder of how much they—and anchoring datasets—can do for you

Before + After view of a zoned flowBefore + After view of a zoned flow

Cheers,
Ashley W.