It would be really helpful to be able to split large project into multiple flows. Currently, combining filters with custom tags sort of enables this, but users need to remember to add the appropriate tag to every dataset and recipe as it's created. It'd be nice if from a project page, I could create multiple flows that separate different aspects of a project. Datasets from neighboring flows could still be accessed in the same way datasets can currently be shared across projects. This would enable a few nice use-cases:
Logically segment long flows into multiple stages for abstraction
Hide preprocessing steps like joining a star schema into a single table to keep the flow from getting too tall
Manage multiple independent but related flows separately
Easily copy an entire flow into multiple versions without needing to manage git branches
Manage ETL flows separately from data preparation flows
Reduce load times for flows - many of my projects take more than 60 seconds to load the flow page
Easily keep track of which datasets apply to each aspect of a project
Optimally, both the flow view and dataset view could be segmented this way.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.