uploading file to S3 bucket

Options
piyushk
piyushk Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Adv Designer, Registered Posts: 55 ✭✭✭✭✭

I have a file and path on S3 bucket is fixed. So, when I upload file to S3 bucket, it automatically appends extra folder to the existing path. This limits it from further auto processing. How can I limit the append of extra path?

Moreover, it uploads file in S3 as out.csv.gz How can I upload file.withy desired file name?

While exporting dataset via export recipe, it creates a folder with random recipe name. How can I restrict from creating the folder?

Answers

  • Alex_Combessie
    Alex_Combessie Alpha Tester, Dataiker Alumni Posts: 539 ✭✭✭✭✭✭✭✭✭
    Options

    Hi,

    Unfortunately, I am unable to reproduce the situation you are describing.

    Could you please detail the different steps by adding some screenshots of your flow and recipe settings?

    Cheers,

    Alex

  • piyushk
    piyushk Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Adv Designer, Registered Posts: 55 ✭✭✭✭✭
    Options

    S3 Connection:

    S3_Connection.jpg

    Content of Path prefix: dataiku_test_db/dataiku_test/snapshot/2020/04/10/

    ------------------------------------------------------------------------------------------------------

    I am exporting a dataset to a fixed path on S3 bucket. So, when I upload file to S3 bucket, DSS automatically appends extra folder to the existing fixed path. This limits it from further auto processing. How can I limit the append of extra path?

    ExtraPath.jpg

    Extra path here added is Orders_Filtered_Demo.

    -------------------------------------------------------------------------------------------------------------------

    DSS uploads file in S3 as out.csv.gz How can I upload file with desired file name like FileName.csv?

    FileName.jpg

    ---------------------------------------------------------------------------------------------------------------------------------------

    While exporting dataset via export recipe, it creates a folder with random recipe name. How can I restrict from creating the folder with random name?

    ExportRecipe.jpgRecipeFolder.jpg

  • Alex_Combessie
    Alex_Combessie Alpha Tester, Dataiker Alumni Posts: 539 ✭✭✭✭✭✭✭✭✭
    Options

    Hi,

    Thanks for the precisions, it helps. I suggest using the "Sync" recipe instead of the "Export recipe".

    After creating the sync recipe, you'll be able to go in the output dataset and edit the settings you are referring to:

    - Path in bucket

    Screenshot 2020-04-16 at 17.36.01.png

    File format / extension

    Screenshot 2020-04-16 at 17.36.09.png

    File name

    Screenshot 2020-04-16 at 17.36.30.png

    Hope it helps,

    Alex

Setup Info
    Tags
      Help me…