While Ingesting new data, is there any code/visual recipe we can update the old records and append new records based on a particular column(metric) value?
In Input dataset, we have 3 records and in the next run we get 3 more records out of which 1 is same as before, 1 is the updated records and 1 is the new record.
In resultset, we want the new record to get append and update the old record and keep the same record as it is.
Hello @sagar_dubey ,
I have tried to add current date column by block, and I have done window recipie with partition by id, and select max date in aggregations :
In the post filter I have set : date is the same as date_max
In the output, I have the data I want :
I don't know if it can help you, maybe there is easier way to solve this use case by script or partitionning...
Intresting case 😄 Do you think my approach would work? This might look a bit odd but I think it is working.
Basically I created 2 excels, named them the same, and I am doing a join on the same file, kind of.... joining the publish excel and excel in the folder I am uploading with the same name, it updated the price for rows that already existed and added new one.
Mateusz
Hello,
I think it works too, but insert time dimension seems to be important in order to select the id's latest version ...(it depends on how data are stored) ?
You are right 🙂 all depends on how real data looks like