Remove duplicate and returns 0 rows in the end
Hello! I`ve been trying to create a python recipe that remove duplicates based in a column and keep the last but when I try to use it in my database it remove every single row in the dataframe.
My code was:
# -*- coding: utf-8 -*- import dataiku import pandas as pd # Recipe inputs df = dataiku.Dataset("OTD_2020-2023").get_dataframe() #DROP duplicates based in `processo` column. df.drop_duplicates(subset=["PROCESSO"], inplace=True) # Recipe outputs out = dataiku.Dataset("OTD_2020-2023").write_with_schema(df)
But I also try to put in another code an IF condition to only execute the drop_duplicate if in an especific column there`s any duplicate.
Does someone that knows what is going on right here?
Best Answer
-
Konstantina Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 25 ✭✭✭✭✭
Hi @impossibletovi
!I've noticed that you are writing your output to the same dataset that you have as input. Could you try to write the output to another dataset and check if the issue is resolved?
You also mentioned that you want to keep the last occurrences of the duplicated records, so you need to add keep='last' in the drop_duplicates() because, by default, it keeps the first occurrences.
-Konstantina
Answers
-
Thank you! I set other place to put the output and it worked.
-
Konstantina Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 25 ✭✭✭✭✭
Glad I could help!