How to append dataframe in existing output dataset
In dataiku v12.3.0, I was trying to append dataframe using write_dataframe() in existing dataset (with same schema). But it always overwrites with last dataframe even though the dataset spec is configured like:
dataset.spec_item["appendMode"] = True
The dataset is classified as output so it doesn't let me use dataset.get_dataframe(). It throws exception: "You cannot read dataset test.my-dataset, it is not declared as an input"
You don't set this in code, you do it on the Inputs/Outputs tab of the recipe and set the "Append instead of overwrite" check box. Then you write normally to the output like in a normal recipe and Dataiku will do the append for you.