Does dataiku has a recipe or something that checks for the changes in the source tables ?

Options

Whenever there is a change in a source table(SQL Server) is there a way to find the schema changes before the project fails? Its a daily run project. I am new to Dataiku so. still weak at understanding the software.

Answers

  • EstelleB
    EstelleB Dataiker, Registered Posts: 6 Dataiker
    Options

    Hello,

    Do you want to check that automatically, on a daily basis? Or is it when designing your flow that you want to check schema changes?

  • Im up for anything. I would like to include this step in already built projects/flows.

    I would like to add this to future flows as well.. Need not be a daily check but whenever there is a change...

  • EstelleB
    EstelleB Dataiker, Registered Posts: 6 Dataiker
    edited July 17
    Options

    In your case, I think that the best way to automate that would be to use a SQL Trigger in a scenario. You can read more there about scenario triggers here: https://doc.dataiku.com/dss/latest/scenarios/triggers.html.

    When you create a scenario, you can choose the trigger called "SQL query change". You then have to add in the code area "SQL Script" something like :

    select column_name, data_type
    from INFORMATION_SCHEMA.COLUMNS where table_name = '<name of table>';

    Where you just have to replace <name of table> by your table name.

    Then you can add any step you need, like "Send message"

    Is that what you need?

Setup Info
    Tags
      Help me…