Going through the hard time while challenging the schema change effect from the sources. can anyone please help on the below query.
i have a partitioned whole dataset (consist the data of all the months, as its partitioned by month) , and on periodic basis (here monthly) i append the monthly dataset received from sources into this whole base dataset but sometimes the input may have different columns names or with few new columns, so when i try load this dataset in to the whole Dataset due to schema change effect either this whole dataset gets deleted or contaminate the columns in schema.
what i am looking for is to have this updated by adding a new column name correctly in the whole dataset creating the null values for the months when that new column is not present and have values for the month it actually uploaded .
