Stale schema in DataIKU flow

allan
allan Registered Posts: 13 ✭✭✭✭

We have a SQL Server backing store that we use to store our data.

A simple SQL recipe created a table that we now need to update the schema on - we're making 2 text fields longer.

I switched the schema to be Manually Define, updated the length of the two fields and ran the flow. The table was created and populated correctly.

When I switch the schema back to Automatically Generate, the flow now has the previous lengths so I assume they are cached somewhere. I see them being used in the logs to create the table.

How can I tell DataIKU to update those cached values based on the data in the current table?

Best Answer

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Answer ✓

    Hi,

    The length is stored in the schema of the dataset. Please go to the "Schema" tab of the dataset settings, select your column, and in the right part of the screen, update the "max length"

Answers

  • allan
    allan Registered Posts: 13 ✭✭✭✭

    @Clément_Stenac

    Thanks, that worked - it does look like that doesn't propagate when we use "Propagate schemas across Flow from here". Is there a way to make that happen, or do we need to adjust the length in the schema for each table individually?

Setup Info
    Tags
      Help me…