uploading file to S3 bucket

piyushk
Level 4
uploading file to S3 bucket

I have a file and path on S3 bucket is fixed. So, when I upload file to S3 bucket, it automatically appends extra folder to the existing path. This limits it from further auto processing. How can I limit the  append of extra path?

Moreover, it uploads file in S3 as out.csv.gz How can I upload file.withy desired file name?

While exporting dataset via export recipe, it creates a folder with random recipe name. How can I restrict from creating the folder?

3 Replies
Alex_Combessie
Dataiker Alumni

Hi,

Unfortunately, I am unable to reproduce the situation you are describing.

Could you please detail the different steps by adding some screenshots of your flow and recipe settings?

Cheers,

Alex

 

 

piyushk
Level 4
Author

S3 Connection:

S3_Connection.jpg

 Content of Path prefix: dataiku_test_db/dataiku_test/snapshot/2020/04/10/

------------------------------------------------------------------------------------------------------

I am exporting a dataset to a fixed path on S3 bucket. So, when I upload file to S3 bucket, DSS automatically appends extra folder to the existing fixed path. This limits it from further auto processing. How can I limit the  append of extra path?

ExtraPath.jpg

 Extra path here added is Orders_Filtered_Demo.

-------------------------------------------------------------------------------------------------------------------

DSS uploads file in S3 as out.csv.gz How can I upload file with desired file name like FileName.csv?

FileName.jpg

---------------------------------------------------------------------------------------------------------------------------------------

While exporting dataset via export recipe, it creates a folder with random recipe name. How can I restrict from creating the folder with random name?

ExportRecipe.jpgRecipeFolder.jpg

 

 

 

Alex_Combessie
Dataiker Alumni

Hi,

Thanks for the precisions, it helps. I suggest using the "Sync" recipe instead of the "Export recipe".

After creating the sync recipe, you'll be able to go in the output dataset and edit the settings you are referring to:

- Path in bucket

Screenshot 2020-04-16 at 17.36.01.png

File format / extension

Screenshot 2020-04-16 at 17.36.09.png

File name

Screenshot 2020-04-16 at 17.36.30.png

Hope it helps,

Alex

0 Kudos