The Dataiku Frontrunner Awards have just launched to recognize your achievements! Submit Your Entry

Write in existing and used folder

Mingo
Level 2
Write in existing and used folder

Hello,

I would like to export all my files created by dataiku ( csv or other ) in a single folder. 

But when I choose to output a recipe in an already existing and already used folder, it doesn't work.

I would prefer not to create a folder per output, is it feasible?

Thanks in advanced

0 Kudos
3 Replies
SarinaS
Dataiker
Dataiker

Hi @Mingo,

Can you explain which type of "output" you are selecting when you attempt to select an output folder that already exists? Are you referring to using an existing "managed folder" when you select your output? 

If so, I think one possible alternative is to configure your output folder at the connection level instead, and use a single filesystem "connection" so that all recipe outputs are to this connection, which is really just a folder on your server/local system. I'll highlight an example of exporting datasets from different projects to the same connection, which all export to the same overall folder.

I have a connection set up on my local system that will write to the folder "my-single-folder":

Screen Shot 2021-03-12 at 1.58.27 PM.png

To illustrate the output, I created sync recipes in two different projects, that both will write to this folder. 

Screen Shot 2021-03-12 at 2.01.30 PM.png

And in another project, just select the same connection:

Screen Shot 2021-03-12 at 2.02.49 PM.png

These will now all land in the same "folder" on my system. If you also need a "managed folder" that points to the same location as this directory, you could then create a managed folder that points to this connection path.  

Screen Shot 2021-03-12 at 1.59.18 PM.png

Let me know if this doesn't address your use case or if you have any questions. 

 

Thanks,

Sarina 

 

 

0 Kudos
Mingo
Level 2
Author

Hello Sarina and thanks for your answer.

I'm using the Export to Folder procedure and I'm creating a qvx file.

So I have to create one object 'folder' , one for each export

Capture1.PNG

At the end I agree, they all end at the same folder on the system but I :

  • I have to create one folder for each output.
  • If I want to change the path of my outputs I have to change it twice (or more)

 

In fact I would like to do this :

 

Capture3.PNG

 And I'm just wondering if it's technically feasible.

 

Thanks

 

 

 

 

 

 

 

 

0 Kudos
SarinaS
Dataiker
Dataiker

Hi @Mingo ,

Thank you for clarifying your use case and posting the screenshots. Indeed, it is not possible to use the same output folder in multiple "Export to folder" recipes.  

Though it would still require all the same initial setup, one option might be to use the "Merge folders" recipe based on all your input folders to consolidate them into 1 final folder:

Screen Shot 2021-03-16 at 4.46.47 PM.png

Thanks,

Sarina 

 

0 Kudos
A banner prompting to get Dataiku DSS