Remove duplicate and returns 0 rows in the end

impossibletovi
impossibletovi Dataiku DSS Core Designer, Registered Posts: 4
edited July 16 in Using Dataiku

Hello! I`ve been trying to create a python recipe that remove duplicates based in a column and keep the last but when I try to use it in my database it remove every single row in the dataframe.

My code was:

# -*- coding: utf-8 -*-
import dataiku
import pandas as pd

# Recipe inputs
df = dataiku.Dataset("OTD_2020-2023").get_dataframe()

#DROP duplicates based in `processo` column.
df.drop_duplicates(subset=["PROCESSO"], inplace=True)

# Recipe outputs
out = dataiku.Dataset("OTD_2020-2023").write_with_schema(df)

But I also try to put in another code an IF condition to only execute the drop_duplicate if in an especific column there`s any duplicate.

Does someone that knows what is going on right here?

Best Answer

  • Konstantina
    Konstantina Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 25 ✭✭✭✭✭
    Answer ✓

    Hi @impossibletovi
    !

    I've noticed that you are writing your output to the same dataset that you have as input. Could you try to write the output to another dataset and check if the issue is resolved?

    You also mentioned that you want to keep the last occurrences of the duplicated records, so you need to add keep='last' in the drop_duplicates() because, by default, it keeps the first occurrences.

    -Konstantina

Answers

Setup Info
    Tags
      Help me…