Reusing flows
Hello I have a question,
We have a set of steps that needs to be performed cross regions. We want to automate everything and have a central point where we can change something and the changes need to be reflected everywhere at once.
I have two questions:
1) Is it possible to link all sub flows together( same flow) and have a change in one reflected in others
2) If I use python spark scripts recipes to do this , are changes in the master recipe reflected in all instances of this recipe across flows.
Best
Best Answer
-
There are no option to "link" recipes together and have changes applied to one transferred to the other. Short of writing the copying logic manually with the public API of DSS, custom recipes (in plugins) or app-as-recipes are the closest to this use case.
Answers
-
Hi,
if you want to design a reusable flow, in the sense of a sequence of recipes applied to a given set of inputs, then you should check https://doc.dataiku.com/dss/latest/applications/index.html#application-as-recipe (there are some limitations, notably around partitioned datasets)
Other solutions would require manually synchronizing recipes between subflow (or automating it with the public API of DSS)
-
Hello,
Thanks for your response, do you have any idea if I were to update the recipe(code or configuration) , then that change would be reflected in places where I used it.
-
Thank you so much. That is exactly what I am looking for.