The February release for the Community is live! Read More

Sync without dropping table

Solved!
khairulfathi
Level 1
Sync without dropping table

Hello,

I'm trying to push processed data to external table (Oracle) as part of data integration. Using Sync recipe, Dataiku will try to drop the table first before loading any data. This is not possible in my case where the Oracle ID I've given does not have DROP privilege. I've tried checking 'Append instead of overwrite' and using Free output schema under settings but both are not working.

Is there any configuration to bypass this or other recipes that can achieve the same outcome?

Thanks for your assistance.

Khairul

0 Kudos
1 Solution
AlexandreV
Dataiker
Dataiker

Hello Khairul, sorry for the delay,

The “Free output schema” should prevent modifications of the schema before the sync recipe to cause change of the schema of the output dataset. If you don’t have the rights to drop table it’s probably good to chose this option.

The “Append instead of overwrite” option should not require a Drop Table of your table if the schema are the same in the input and output of the sync.

Can you confirm that the schema of the two datasets (input and output of sync recipe) are equal?

If that’s the case can you provide the logs of the sync job before the DROP TABLE statement log?

ie few lines before:

[INFO] [dku.dataset.sql] - DROP TABLE "PROJECT_KEY_Table_name"

 

Alex

View solution in original post

0 Kudos
2 Replies
AlexandreV
Dataiker
Dataiker

Hello Khairul, sorry for the delay,

The “Free output schema” should prevent modifications of the schema before the sync recipe to cause change of the schema of the output dataset. If you don’t have the rights to drop table it’s probably good to chose this option.

The “Append instead of overwrite” option should not require a Drop Table of your table if the schema are the same in the input and output of the sync.

Can you confirm that the schema of the two datasets (input and output of sync recipe) are equal?

If that’s the case can you provide the logs of the sync job before the DROP TABLE statement log?

ie few lines before:

[INFO] [dku.dataset.sql] - DROP TABLE "PROJECT_KEY_Table_name"

 

Alex

View solution in original post

0 Kudos
khairulfathi
Level 1
Author

Thank you @AlexandreV for pointing to the right direction. Turns out there's a typo in one of the target column name and the column order got switched. Once fixed, all working perfectly now.

0 Kudos
A banner prompting to get Dataiku DSS