You now have until September 15th to submit your use case or success story to the 2022 Dataiku Frontrunner Awards!ENTER YOUR SUBMISSION

Source input excel schema changes

Solved!
IrinaS
Level 1
Source input excel schema changes

Hi,

I'm loading excel files from FTP to SQL db, everything works fine but I want the build to be stopped if the excel structure changes (missing a column, adding a new column ) and the new schema not to be loaded in the db.

 

I've tried with sync recipe, but If a column is missing the data is loaded automatically from the next columns.

 

Is there a way to obtain this?

 

Thanks,

Irina

1 Solution
AlexT
Dataiker
Dataiker

Hi @IrinaS,

There  2 ways to handle schema for sync recipes, using "Free output schema"  then it will perform named-based. That may help to some extent in your case depending on the type of changes.  

Screenshot 2022-06-09 at 21.51.23.png

If that doesn't help,  you can use Scenario to build these datasets and use the "check flow consistency step" and not proceed to build but instead alert in case the schema is different from the data.

https://knowledge.dataiku.com/latest/courses/flow-views-and-actions/schema-propagation-concept-summa...

Let me know if that helps!

 

View solution in original post

0 Kudos
1 Reply
AlexT
Dataiker
Dataiker

Hi @IrinaS,

There  2 ways to handle schema for sync recipes, using "Free output schema"  then it will perform named-based. That may help to some extent in your case depending on the type of changes.  

Screenshot 2022-06-09 at 21.51.23.png

If that doesn't help,  you can use Scenario to build these datasets and use the "check flow consistency step" and not proceed to build but instead alert in case the schema is different from the data.

https://knowledge.dataiku.com/latest/courses/flow-views-and-actions/schema-propagation-concept-summa...

Let me know if that helps!

 

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku