Bring back flow filters
Older versions of Dataiku had an excellent filter system:
You could save filters and hide unrelated elements, essentially preserving a view of the flow that was minimal, even in a complicated project. There was also an advanced mode where you could build your own expressions, which we used extensively.
In more recent versions of Dataiku, this system was replaced with a simplified search bar:
This UI is definitely a cleaner way to search, but it has dropped the ability to completely filter out the majority of the flow.
For projects of our scale, this is leading us to do complicated workarounds, e.g. creating a new ephemeral project, copying our authoritative flows there, doing some work on them, then copying them back.
If you could bring back that "hide unrelated elements" option and provide a full filter experience, especially if this were available with full api support for managing the preset filters, and we could set a global default filter for the project so users never have to load the full flow, that would be incredible.
You might be wondering why build a project this large? The main reason is versioning and consistency. We want to avoid duplicated pipelines as many users collaborate, and we want to have one versioned object that represents the current manifest of all pipelines. When pipelines get moved into many different projects, it results in needing to keep track of compatibility across all those projects, which becomes a harder task than just maintaining all the pipelines in a single collaborative project with branches and approvals. This should work great, but the Dataiku experience becomes slower at this scale, particularly rendering the flow and populating the dataset selector menus in new dataset creation modals. Once loaded, the framerate of the experience is also low, and the cognitive complexity is high for users to navigate. Filters would solve most of these issues, and a combination of caching and a more efficient rendering approach would probably get it to 100% usable at this scale.
I've been able to use the Dataiku API to produce a much more navigable visualization of the flow at this scale - the endpoints themselves respond very quickly, it's just the native UI that needs filtering and possibly a few other enhancements to make this scale work.
Comments
-
Hello, Thank you for your feedback.
We've added it to our feature backlog.
Regards
