Use a managed folder as output of several recipes ?

MGad
MGad Registered Posts: 14 ✭✭✭✭

Hello,

I have a managed folder containing several result files from an external API. I would like to call this API several times in different recipes but store the results in the same folder each time. Is it doable ?

Thx for your help

Best Answer

  • Liev
    Liev Dataiker Alumni Posts: 176 ✭✭✭✭✭✭✭✭
    edited July 17 Answer ✓

    Hi @MGad

    In the flow indeed DSS won't allow you to put as an output an existing folder where this folder is already the output of another recipe. We do this because it could lead to conflicts and confusion.

    However, this can still be achieved. If you already have a folder on your flow, get the folder id from the URL path when opening the folder. The path will be something like

    DSS_HOST/projects/projectKey/managedfolder/folderId/view/

    Then, in your flow click on +Dataset > Folder to create a new folder. You will need to match the connection to that of the folder we will mirror. Then in Settings > Storage > Path replace the ${odbId} part with the folderId from above.

    From a DSS perspective, these will be two distinct folders and you will be able to have them as outputs of different recipes. But if you look at the contents you'll see they're the same as they point to the same location.

    Keep in mind what I mentioned at the beginning of the post re keeping your design clear as otherwise, it might lead to issues in your project later on.

    I hope this helps!

Answers

  • MGad
    MGad Registered Posts: 14 ✭✭✭✭

    Alright, thank you for your clear answer

  • Tanguy
    Tanguy Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2023 Posts: 118 Neuron

    I was wondering: what does "odbId" stands for (in the folder path "${projectKey}/${odbId}")?

  • Prostko
    Prostko Partner, Registered Posts: 1 Partner

    I believe it stands for Operational Database (ODB) Identification (ID).

    I am rationalizing 'Operational' instead of 'Object', 'Open', or 'Online' because an operational database is a database that is used to manage and store data in real-time.

Setup Info
    Tags
      Help me…