Unable to write to dataset - TypeError: 'ObjectBlock' object is not iterable

[emher]

When i try to write a pandas dataframe to a dataset, i.e.

df = pd.DataFrame(...)
ds = dataiku.Dataset(...)
ds.write_with_schema(df, dropAndCreate=True)

i get the following error,

TypeError: 'ObjectBlock' object is not iterable

Have anyone tried something similar? And/or do you know what might be going wrong? Inspecting the dataset, I can see that the schema is written as intended, but no data is written.

EDIT: The error occurs only when I call the code from outside dataiku, if I create a recipe inside dataiku, it works as intended.

[fchataigner2]

Hi,

if it works in DSS, then maybe the cause is a discrepancy in package versions, notably of Pandas and/or Numpy. Can you get a `pip list` of the python environment where you get this error. Also, what's the full stacktrace of the error? (ie where is this error raised from)

[emher]

The full stack trace is,

Traceback (most recent call last):   File "/home/emher/Projects/pipeline_test/PFC_python_library/tmp.py", line 35, in <module>     pfc.delivery.deliver(df_ecmwf, df_pfc, row.to_dict())   File "/home/emher/Projects/pipeline_test/PFC_python_library/pfc/delivery.py", line 330, in deliver     log_handler(meta_data, delivery_blob, error)   File "/home/emher/Projects/pipeline_test/PFC_python_library/tmp.py", line 31, in <lambda>     pfc.delivery.log_handler = lambda x, y, z: log_to_dataset(x, y, z, dataset=dataiku.Dataset("delivery_logs"))   File "/home/emher/Projects/pipeline_test/PFC_python_library/pfc/delivery.py", line 402, in log_to_dataset     writer.write_dataframe(df_log)   File "/home/emher/Projects/pipeline_test/PFC_python_library/venv/lib/python3.8/site-packages/dataiku/core/dataset_write.py", line 395, in write_dataframe     dku_pandas_csv.DKUCSVFormatter(df, self.remote_writer,   File "/home/emher/Projects/pipeline_test/PFC_python_library/venv/lib/python3.8/site-packages/dataiku/core/dku_pandas_csv.py", line 197, in save     self._save()   File "/home/emher/Projects/pipeline_test/PFC_python_library/venv/lib/python3.8/site-packages/dataiku/core/dku_pandas_csv.py", line 296, in _save     self._save_chunk(start_i, end_i)   File "/home/emher/Projects/pipeline_test/PFC_python_library/venv/lib/python3.8/site-packages/dataiku/core/dku_pandas_csv.py", line 318, in _save_chunk     for col_loc, col in zip(b.mgr_locs, d): TypeError: 'ObjectBlock' object is not iterable 0 rows successfully written (NBRAVMN4TD) 

Hence the error arises within dku_pandas_csv. Locally, my numpy and pandas versions are,

numpy==1.20.0
pandas==1.2.1

I am not sure what version dataiku uses internally. Do you know where i can see this?

[fchataigner2]

Hi,

you're indeed using a very recent Pandas, and python 3.7. DSS' python code actually doesn't handle it yet, so you should revert your pandas to pandas>=1.0,<1.1, and possibly use a python3.6 if that doesn't solve the error