Export dataset to CSV with variable filename

Options
Filip_Pejsa
Filip_Pejsa Registered Posts: 11 ✭✭✭

Hi,

Is there a simple, non-coding way to export a dataset to a folder (via export recipe) under a file name reflecting a project variable value ?

E.g. the source dataset name is "output", the project variable is {"filedate": "30May2023"} , the exported file in the folder should be output_${filedate}, i.e. "output_30May2023.csv".

Thanks for advice.


Operating system used: Windows 10

Answers

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    edited July 17
    Options

    Hi @Filip_Pejsa
    ,

    The Export to Folder recipe does use the input dataset name as the exported filename for any files exported to the folder.

    Screenshot 2023-06-02 at 4.05.43 PM.png

    If you need to specify the export file name based on a project variable, you would have to use code to do this. However, you could create a quite simple recipe that does this and turn it into a plugin, which would allow for reuse of the plugin in the form a visual recipe for all other users.

    Here is an example

    # -*- coding: utf-8 -*-
    import dataiku
    import pandas as pd, numpy as np
    from dataiku import pandasutils as pdu
    
    # Read recipe inputs
    Input_data = dataiku.Dataset("<INPUT_DATASET>")
    Input_data_df = Input_data.get_dataframe()
    
    client = dataiku.api_client()
    project = client.get_project('<PROJECT_ID>')
    variables = project.get_variables()
    filedate = variables['standard']['filedate']
    filename = 'output_' + filedate + '.csv'
    
    # Write recipe outputs
    outputfolder = dataiku.Folder("<FOLDERID>")
    outputfolder_info = outputfolder.get_info()
    
    with outputfolder.get_writer(filename) as w:
        w.write(Input_data_df.to_csv().encode())


    You can then convert this recipe to a plugin with the following option from a Python recipe:

    Screenshot 2023-06-02 at 4.21.05 PM.png

    You will need to modify the plugin parameters and input and output roles, and it might help to review the documentation on this. Once you have a plugin, users can select your plugin recipe from the flow.

    Thank you,
    Sarina 

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,595 Neuron
    Options

    @SarinaS
    ,

    I’m looking forward to seeing how this works out.

Setup Info
    Tags
      Help me…