Transfer file to server with timestamp

Solved!
tripathirahul
Level 2
Transfer file to server with timestamp

Hi Team,

 

I'm trying to upload csv with timestamp attached to it to server using sftp protocol but after trying all suggessted ways like partition, python receipe but none is working for me.

Flow

MLoutput - Snowflake - blob storage - server

Regards

Rahul

 

 

 

 

0 Kudos
1 Solution
VitaliyD
Dataiker

 Hi Rahul,

The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
dataset = dataiku.Dataset("dataset_name")
df = dataset.get_dataframe()

#generate your custom name with your own logic
my_custom_name = "my_custom_name1.csv"

# Write recipe outputs to the sftp managed folder
sftp_folder = dataiku.Folder("folder_id")
sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))

Hope this helps.

Best,

Vitaliy

View solution in original post

0 Kudos
6 Replies
VitaliyD
Dataiker

Hi,

Please provide more details so we can better understand the issue:
1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file.
2) What output do you get?
3) What is desired output?

Best,

Vitaliy

0 Kudos
tripathirahul
Level 2
Author

1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file. - Yes
2) What output do you get? - out-s0.csv
3) What is desired output? - file must be save like this market_name_project_name_timestamp.csv example za_offer_recommendation_2017-07-01 12:00:00.000.csv

Files and Folders are also getting deleted in server after  every run

 

Regards

Rahul

0 Kudos
VitaliyD
Dataiker

There is no such functionality to set the file name automatically. You can do it manually from the dataset Settings > Advanced page by Forcing it to a single output file and specifying the file name.

Screenshot 2022-10-15 at 17.27.09.png

Otherwise, you can create a managed folder and then use a python recipe to read the dataset into a data frame and then generate the filename and save the data frame as a CSV file into the folder.

I hope this helps.
Best,

Vitaliy

0 Kudos
tripathirahul
Level 2
Author

Hi Vitaliy,

using below python code i'm able to generate file in managed folder but when i'm trying to sync to server using sync receipe it is still saving as out-s0.csv. could you please let me know what i'm doing wrong here

LInk - https://community.dataiku.com/t5/Plugins-Extending-Dataiku/Exporting-a-file-in-dynamic-name/td-p/107...

Flow - PFA

Regards

Rahul

 

 

Regards

Rahul

0 Kudos
VitaliyD
Dataiker

 Hi Rahul,

The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
dataset = dataiku.Dataset("dataset_name")
df = dataset.get_dataframe()

#generate your custom name with your own logic
my_custom_name = "my_custom_name1.csv"

# Write recipe outputs to the sftp managed folder
sftp_folder = dataiku.Folder("folder_id")
sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))

Hope this helps.

Best,

Vitaliy

0 Kudos
tripathirahul
Level 2
Author

Hi VitaliyD,

Thanks, now I am able to upload files in sftp server

 

 

Regards

Rahul

0 Kudos