We're excited to announce that we're launching the second installment of Dataiku Product Days Register Now

Merge Sync, Filter, and Sort into a single recipe

0 Kudos

The Sort recipe can already do everything Sync and Filter can, but Sync cannot Filter and Filter cannot Sort. Team members will often start by just using Sync or Filter, but may eventually need Filter or Sort later on, and end up replacing the recipe. I've personally done this 30-40 times already before making a rule for myself to always use Filter or Sort (depending on whether the database type supports ordered datasets) and never Sync, since that gives the most flexibility.

My idea is to merge these recipes together into one, so that users have the flexibility to effectively edit the WHERE and ORDER BY clauses of their SQL at any time even if they didn't foresee needing them at recipe creation.

The Top and Distinct recipes could also potentially be merged into this new super-recipe. 

This would simplify the UI and avoid recipe replacements or duplicate datasets being created.