Table overwritten instead of append due to schema inconsistency

kuraigal_dss Registered Posts: 2 ✭✭✭

I'm trying to append data instead of overwrite with dataset including decimal field values, Standard in target DB is to define Decimal as column datatype.

In this case, Dataiku by default defines datatype of this field as Float or Double in schema, causing the table to be dropped and re-create due to schema inconsistency

Is there any way to overcome this, I would like to retain the table and the data in it without having to create additional transient table

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Answer ✓


    DSS has no mapping for the DECIMAL type, so at the moment it will always consider the table schema as incompatible and attempt to recreate the table. Using a temporary table for flushing in the other table is a good solution.


  • kuraigal_dss
    kuraigal_dss Registered Posts: 2 ✭✭✭

    Thanks, I tried the above approach and it worked.

    Additionally I ended up using sql script instead of sql query recipe to load from the temporary table to final bypassing schema compatibility check

Setup Info
      Help me…