Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
The error suggests you are using code that writes to the local filesystem.
For non-filesystem managed folders (HDFS, S3, โฆ), you need to use the various read/download and write/upload APIs.
For example use upload_stream() or upload_file() SeeL https://doc.dataiku.com/dss/latest/python-api/managed_folders.html for more details.
Here is an generic example :
```
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
managed_folder_id = "URKU7Oqb"
# Read dataset convert df to csv inst
my_dataset = dataiku.Dataset("customers_labeled_prepared")
df = my_dataset.get_dataframe()
df.to_csv(index=False).encode("utf-8")
# Write recipe outputs
output_folder = dataiku.Folder(managed_folder_id)
output_folder.upload_stream("some_name.csv", df.to_csv(index=False).encode("utf-8"))
```
Hi Ankur,
By default, any new data will be written to new files when syncing to an S3 Dataset.
To change this behavior you can edit the settings of the output dataset under Advanced - Force single output file and you can also set the file base name :
Please refer to screenshot below:
Let me know if that works for you.
Hi @AlexT ,
Thanks for this but I want to write all the input DSS datasets in the csv format to my s3 bucket using python recipe. But while writing I am getting error. Attached is the screenshot of error message.
Regards,
Ankur.
Hi,
The error suggests you are using code that writes to the local filesystem.
For non-filesystem managed folders (HDFS, S3, โฆ), you need to use the various read/download and write/upload APIs.
For example use upload_stream() or upload_file() SeeL https://doc.dataiku.com/dss/latest/python-api/managed_folders.html for more details.
Here is an generic example :
```
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
managed_folder_id = "URKU7Oqb"
# Read dataset convert df to csv inst
my_dataset = dataiku.Dataset("customers_labeled_prepared")
df = my_dataset.get_dataframe()
df.to_csv(index=False).encode("utf-8")
# Write recipe outputs
output_folder = dataiku.Folder(managed_folder_id)
output_folder.upload_stream("some_name.csv", df.to_csv(index=False).encode("utf-8"))
```
Hi @AlexT ,
Thank you for all the help and support you have provided to me till now. Looking forward for your continued support. i really appreciate it.
Thank You,
Ankur.