Whenever there is a change in a source table(SQL Server) is there a way to find the schema changes before the project fails? Its a daily run project. I am new to Dataiku so. still weak at understanding the software.
Do you want to check that automatically, on a daily basis? Or is it when designing your flow that you want to check schema changes?
Im up for anything. I would like to include this step in already built projects/flows.
I would like to add this to future flows as well.. Need not be a daily check but whenever there is a change...
In your case, I think that the best way to automate that would be to use a SQL Trigger in a scenario. You can read more there about scenario triggers here: https://doc.dataiku.com/dss/latest/scenarios/triggers.html.
When you create a scenario, you can choose the trigger called "SQL query change". You then have to add in the code area "SQL Script" something like :
select column_name, data_type from INFORMATION_SCHEMA.COLUMNS where table_name = '<name of table>';
Where you just have to replace <name of table> by your table name.
Then you can add any step you need, like "Send message" 😉
Is that what you need?