Use a managed folder as output of several recipes ?
Hello,
I have a managed folder containing several result files from an external API. I would like to call this API several times in different recipes but store the results in the same folder each time. Is it doable ?
Thx for your help
Best Answer
-
Hi @MGad
In the flow indeed DSS won't allow you to put as an output an existing folder where this folder is already the output of another recipe. We do this because it could lead to conflicts and confusion.
However, this can still be achieved. If you already have a folder on your flow, get the folder id from the URL path when opening the folder. The path will be something like
DSS_HOST/projects/projectKey/managedfolder/folderId/view/
Then, in your flow click on +Dataset > Folder to create a new folder. You will need to match the connection to that of the folder we will mirror. Then in Settings > Storage > Path replace the ${odbId} part with the folderId from above.
From a DSS perspective, these will be two distinct folders and you will be able to have them as outputs of different recipes. But if you look at the contents you'll see they're the same as they point to the same location.
Keep in mind what I mentioned at the beginning of the post re keeping your design clear as otherwise, it might lead to issues in your project later on.
I hope this helps!
Answers
-
Alright, thank you for your clear answer
-
Tanguy Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2023 Posts: 124 Neuron
I was wondering: what does "odbId" stands for (in the folder path "${projectKey}/${odbId}")?
-
I believe it stands for Operational Database (ODB) Identification (ID).
I am rationalizing 'Operational' instead of 'Object', 'Open', or 'Online' because an operational database is a database that is used to manage and store data in real-time.