Failed to synchronize Hive Metastore for dataset error

jrod31

Hi,

I am running 3 flows under the same project with the same prepare recipe and I get the following error message on the 3rd flow and not on the other 2 flow.

HIVE_METASTORE_SYNCHRONIZE occurred 1 times
Failed to synchronize Hive metastore for dataset

The source of the dataset are CSV files from a SharePoint.
The flow is setup as follow:

- The dataset is created based on several CSV files on a SharePoint
- A prepare recipe is added (a column rename recipe) and I get the error after running it.

I run it on Local Stream.
The other 2 flows run good without any errors.
I have seen other posts but can't seem to figure it out.

Any idea or thoughts on this issue?

Thanks,

---

Operating system used: Windows

vamsikeka

Go to settings of the target dataset and uncheck the "Auto-sync to Hive metastore". In case if you want to create the dataset as table in the backend Hive DB, then make sure whatever the column names and datatypes in prepare recipe are inline with the output dataset schema as well as the Hive table structure which that dataset is loading.

Thanks,

Vamsi

jrod31

@vamsikeka
This works!

Any thoughts why I would need to disable this in 1 flow and not the other 2?

Thanks,

vamsikeka

When you create a recipe, it will automatically generate backend schema. May be the other datasets are already in line in terms of table structure. And it is best to uncheck that hive metastore check box unless it is necessary. Else it will create lot of unnecessary tables in the backend.

Thanks

Vamsi

jrod31

Thanks!