why I can't overwritte a table from dataiku

hbemarion

Hello everybody,

I have a table "my_table" in my database. I import my_table in dataiku and I realize an outer join to complete the value from two columns in my_table. In output of the outer_join, I create a file system.

After that, I used the "sync" recipe to put the result of the join in "my_table". The "sync" recipe run without error, but I have no modification in my_table.

I don't understand why. Dataiku have the grant in my database to realize an update, insert, truncate… Commit are automatic. I have the same schema…

Is there any right to add somewhere to overwritte a table ?

Thank you

Turribeach

While Dataiku can do ETL work it's not an ETL tool. Therefore some familiar concepts from ETL tools don't apply in Dataiku. In general you should not write back to input tables, Dataiku works on the concept that for every transformation (recipe) you should have a new output table and you should be able to see the intermediate results. This helps less technical users understand how data is transformed in your flow (your data pipeline). Your design breaks this concept for clear benefit. Furthermore moving data across different connection types is inherently inneficient and will likely lead to performance issues. So if you connection allows to write back then simply create a new dataset with the joined results and use that going forward.