Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi everyone,
I am trying to compress a CSV and upload to sharepoint. However, I find that csv file is not actually compressed - also, when I look at the preview of sharepoint folder in dataiku - it shows the following error.
I used the code similar to what has been shared in the post and solution. Any suggestions are welcome.
I use enterprise version - Dataiku 9.0.3. Is there a limit on size of the file that could be uploaded to sharepoint through dataiku?
Double posting the answer here for future reference:
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
import io
import dataikuapi
from datetime import datetime
import gzip
input_data = dataiku.Dataset("input_data")
input_data_df = input_data.get_dataframe()
content = input_data_df.to_csv(index=False).encode("utf-8")
compressed_content = gzip.compress(content)
output_folder = dataiku.Folder("output_data")
output_folder.upload_stream("filename.csv.gz", compressed_content)
Double posting the answer here for future reference:
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu
import io
import dataikuapi
from datetime import datetime
import gzip
input_data = dataiku.Dataset("input_data")
input_data_df = input_data.get_dataframe()
content = input_data_df.to_csv(index=False).encode("utf-8")
compressed_content = gzip.compress(content)
output_folder = dataiku.Folder("output_data")
output_folder.upload_stream("filename.csv.gz", compressed_content)