Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi Rahul,
The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
# Read recipe inputs
dataset = dataiku.Dataset("dataset_name")
df = dataset.get_dataframe()
#generate your custom name with your own logic
my_custom_name = "my_custom_name1.csv"
# Write recipe outputs to the sftp managed folder
sftp_folder = dataiku.Folder("folder_id")
sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))
Hope this helps.
Best,
Vitaliy
Hi,
Please provide more details so we can better understand the issue:
1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file.
2) What output do you get?
3) What is desired output?
Best,
Vitaliy
1) What input data timestamp values do you have? Is it something like "2017-07-01 12:00:00.000" or something else? It would be even better if you could attach a sample file. - Yes
2) What output do you get? - out-s0.csv
3) What is desired output? - file must be save like this market_name_project_name_timestamp.csv example za_offer_recommendation_2017-07-01 12:00:00.000.csv
Files and Folders are also getting deleted in server after every run
Regards
Rahul
There is no such functionality to set the file name automatically. You can do it manually from the dataset Settings > Advanced page by Forcing it to a single output file and specifying the file name.
Otherwise, you can create a managed folder and then use a python recipe to read the dataset into a data frame and then generate the filename and save the data frame as a CSV file into the folder.
I hope this helps.
Best,
Vitaliy
Hi Vitaliy,
using below python code i'm able to generate file in managed folder but when i'm trying to sync to server using sync receipe it is still saving as out-s0.csv. could you please let me know what i'm doing wrong here
Flow - PFA
Regards
Rahul
Regards
Rahul
Hi Rahul,
The sync recipe won't work, as there is no control over the filenames produced by visual recipes (unless you do it manually, as I explained earlier). So instead of using the sync recipe, you can create a managed folder using the sftp connection and save files directly from the python recipe into the managed folder using the upload_stream() method. Please refer to the code example below:
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
# Read recipe inputs
dataset = dataiku.Dataset("dataset_name")
df = dataset.get_dataframe()
#generate your custom name with your own logic
my_custom_name = "my_custom_name1.csv"
# Write recipe outputs to the sftp managed folder
sftp_folder = dataiku.Folder("folder_id")
sftp_folder.upload_stream(my_custom_name, df.to_csv(index=False).encode("utf-8"))
Hope this helps.
Best,
Vitaliy
Hi VitaliyD,
Thanks, now I am able to upload files in sftp server
Regards
Rahul