Write in existing and used folder

Options
Mingo
Mingo Registered Posts: 7 ✭✭✭
in Using Dataiku

Hello,

I would like to export all my files created by dataiku ( csv or other ) in a single folder.

But when I choose to output a recipe in an already existing and already used folder, it doesn't work.

I would prefer not to create a folder per output, is it feasible?

Thanks in advanced

· Share on FacebookShare on Twitter

Answers

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    Options

    Hi @Mingo
    ,

    Can you explain which type of "output" you are selecting when you attempt to select an output folder that already exists? Are you referring to using an existing "managed folder" when you select your output?

    If so, I think one possible alternative is to configure your output folder at the connection level instead, and use a single filesystem "connection" so that all recipe outputs are to this connection, which is really just a folder on your server/local system. I'll highlight an example of exporting datasets from different projects to the same connection, which all export to the same overall folder.

    I have a connection set up on my local system that will write to the folder "my-single-folder":

    Screen Shot 2021-03-12 at 1.58.27 PM.png 

    To illustrate the output, I created sync recipes in two different projects, that both will write to this folder.

    Screen Shot 2021-03-12 at 2.01.30 PM.png

    And in another project, just select the same connection:

    Screen Shot 2021-03-12 at 2.02.49 PM.png

     

     

    These will now all land in the same "folder" on my system. If you also need a "managed folder" that points to the same location as this directory, you could then create a managed folder that points to this connection path.

    Screen Shot 2021-03-12 at 1.59.18 PM.png

    Let me know if this doesn't address your use case or if you have any questions.

    Thanks,

    Sarina 

    · Share on FacebookShare on Twitter
  • Mingo
    Mingo Registered Posts: 7 ✭✭✭
    Options

    Hello Sarina and thanks for your answer.

    I'm using the Export to Folder procedure and I'm creating a qvx file.

    So I have to create one object 'folder' , one for each export

    Capture1.PNG

    At the end I agree, they all end at the same folder on the system but I :

    • I have to create one folder for each output.
    • If I want to change the path of my outputs I have to change it twice (or more)

    In fact I would like to do this :

    Capture3.PNG

     And I'm just wondering if it's technically feasible.

    Thanks

    · Share on FacebookShare on Twitter
  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    Options

    Hi @Mingo
    ,

    Thank you for clarifying your use case and posting the screenshots. Indeed, it is not possible to use the same output folder in multiple "Export to folder" recipes.

    Though it would still require all the same initial setup, one option might be to use the "Merge folders" recipe based on all your input folders to consolidate them into 1 final folder:

    Screen Shot 2021-03-16 at 4.46.47 PM.png

    Thanks,

    Sarina 

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…