Transfer file to server with timestamp

Options
tripathirahul
tripathirahul Registered Posts: 4

Hi Team,

I'm trying to upload csv with timestamp attached to it to server using sftp protocol but after trying all suggessted ways like partition, python receipe but none is working for me.

Flow

MLoutput - Snowflake - blob storage - server

Regards

Rahul

Tagged:

Best Answer

  • VitaliyD
    VitaliyD Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 102 Dataiker
    edited July 17 Answer ✓
    Options

    Hi Rahul,

    The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:

    import dataiku
    import pandas as pd, numpy as np
    from dataiku import pandasutils as pdu
    
    # Read recipe inputs
    dataset = dataiku.Dataset("dataset_name")
    df = dataset.get_dataframe()
    
    #generate your custom name with your own logic
    my_custom_name = "my_custom_name1.csv"
    
    # Write recipe outputs to the sftp managed folder
    sftp_folder = dataiku.Folder("folder_id")
    sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))

    Hope this helps.

    Best,

    Vitaliy

Answers

  • VitaliyD
    VitaliyD Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 102 Dataiker
    Options

    Hi,

    Please provide more details so we can better understand the issue:
    1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file.
    2) What output do you get?
    3) What is desired output?

    Best,

    Vitaliy

  • tripathirahul
    tripathirahul Registered Posts: 4
    Options

    1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file. - Yes
    2) What output do you get? - out-s0.csv
    3) What is desired output? - file must be save like this market_name_project_name_timestamp.csv example za_offer_recommendation_2017-07-01 12:00:00.000.csv

    Files and Folders are also getting deleted in server after every run

    Regards

    Rahul

  • VitaliyD
    VitaliyD Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 102 Dataiker
    Options

    There is no such functionality to set the file name automatically. You can do it manually from the dataset Settings > Advanced page by Forcing it to a single output file and specifying the file name.

    Screenshot 2022-10-15 at 17.27.09.png

    Otherwise, you can create a managed folder and then use a python recipe to read the dataset into a data frame and then generate the filename and save the data frame as a CSV file into the folder.

    I hope this helps.
    Best,

    Vitaliy

  • tripathirahul
    tripathirahul Registered Posts: 4
    Options

    Hi Vitaliy,

    using below python code i'm able to generate file in managed folder but when i'm trying to sync to server using sync receipe it is still saving as out-s0.csv. could you please let me know what i'm doing wrong here

    LInk - https://community.dataiku.com/t5/Plugins-Extending-Dataiku/Exporting-a-file-in-dynamic-name/td-p/10756

    Flow - PFA

    Regards

    Rahul

    Regards

    Rahul

  • tripathirahul
    tripathirahul Registered Posts: 4
    Options

    Hi VitaliyD,

    Thanks, now I am able to upload files in sftp server

    Regards

    Rahul

Setup Info
    Tags
      Help me…