Schema errors while using the Chunked writing built in fucntionality

I want to split a dataframe of 4000 columns in 5 diffrent tables to be able to write it back to SQL.

However, our instance is strugling with writing a table of 1000 columns to schema, thus I had to obtain a writer as per DSS docs.

However, while the writer is working for the first round it crushes on the second one raising a schema error which we cannot really crack.

Could you please help on what goes wrong? Why the schema of the empty table is not been updated based on the input that is about to be writen?

Thanks in advance,
A

sql_datasets= [
  "DTO_Risk_Factors_unshifted_1",
"DTO_Risk_Factors_unshifted_2"]

inp = dataiku.Dataset("DLS_TEST.DTO_Risk_Factors")

start = 0
end = 950
step = 950


for out in  sql_datasets:

    print ('starting location', start)
    print ('ending location', end)
    print(out)

    out = dataiku.Dataset(out)
    
    with out.get_writer() as writer:
        inp = dataiku.Dataset("DLS_TEST.DTO_Risk_Factors")
        for df in inp.iter_dataframes( chunksize=10500):
            print (len(df))
            # preprocess\

            df_temp = df.iloc[:, start:end ]
            print(df_temp.shape)
            
            # Write the processed dataframe
            writer.write_dataframe(df_temp)
            
    start +=  step
    end +=  step