Transfer file to server with timestamp

tripathirahul

Hi Team,

I'm trying to upload csv with timestamp attached to it to server using sftp protocol but after trying all suggessted ways like partition, python receipe but none is working for me.

Flow

MLoutput - Snowflake - blob storage - server

Regards

Rahul

Vitaliy

Hi Rahul,

The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
dataset = dataiku.Dataset("dataset_name")
df = dataset.get_dataframe()

#generate your custom name with your own logic
my_custom_name = "my_custom_name1.csv"

# Write recipe outputs to the sftp managed folder
sftp_folder = dataiku.Folder("folder_id")
sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))

Hope this helps.

Best,

Vitaliy

Vitaliy

Hi,

Please provide more details so we can better understand the issue:
1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file.
2) What output do you get?
3) What is desired output?

Best,

Vitaliy

tripathirahul

1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file. - Yes
2) What output do you get? - out-s0.csv
3) What is desired output? - file must be save like this market_name_project_name_timestamp.csv example za_offer_recommendation_2017-07-01 12:00:00.000.csv

Files and Folders are also getting deleted in server after every run

Regards

Rahul

Vitaliy

There is no such functionality to set the file name automatically. You can do it manually from the dataset Settings > Advanced page by Forcing it to a single output file and specifying the file name.

Otherwise, you can create a managed folder and then use a python recipe to read the dataset into a data frame and then generate the filename and save the data frame as a CSV file into the folder.

I hope this helps.
Best,

Vitaliy

tripathirahul

Hi Vitaliy,

using below python code i'm able to generate file in managed folder but when i'm trying to sync to server using sync receipe it is still saving as out-s0.csv. could you please let me know what i'm doing wrong here

LInk - https://community.dataiku.com/t5/Plugins-Extending-Dataiku/Exporting-a-file-in-dynamic-name/td-p/10756

Flow - PFA

Regards

Rahul

Regards

Rahul

tripathirahul

Hi VitaliyD,

Thanks, now I am able to upload files in sftp server

Regards

Rahul