InvocationTargetException while writing dataset into postgreSql DB.

srikanth_ch

Hi Team,

I am new to DataIKU and when I try to execute one of the python recipe which stores dataframe into Postgresql table it throws exception .

Exception trace says, duplicate column, but the table is empty.

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_markit_rejections"
Detail: Key (business_area, commodity_name, validation_date, team_name, trading_hub_full_name, trading_hub, ac_symbol, term_type, term_year, term_start_date)=(GPTA, Power, 2023-05-08, East, ZZZZZZ - 5x16 / 2x16 / 7x8, YYYYYYY - 5x16 / 2x16 / 7x8 - Off, XXXXXXXX, Month, 2023, 2023-09-01) already exists.
Where: COPY markit_rejections, line 2164

---

Operating system used: Windows 11

Alexandru

Hi @srikanth_ch
,

It looks like you are trying to write to an existing table or table created with a custom SQL creation script?
As typically, this wouldn't come up when using a managed SQL dataset in DSS.

To resolve this issue, you could either modify the data that is being inserted to ensure that the primary key values are unique or remove the primary key constraint to allow for duplicates.

Thanks

srikanth_ch

Hi @AlexT

Thank you for the response. Yes, I am trying to write into existing table, but the record which I am inserting is not present in the table. The insertion record is unique.

xyz_dataset= dataiku.Dataset("xyz")
xyz_dataset.write_with_schema(abc_df)

Alexandru

The error would suggest the record is already present so hard to explain why you would be seeing this. I would suggest you open a support ticket with job diagnostics so we can investigate further.

Have you tried writing to a new dataset/table managed by DSS are you seeing any issues. When writing from DSS its expected you use managed dataset and not existing tables, existing tables should be used as input datasets.

srikanth_ch

Thanks @AlexT

I raised this concern to internal dataIKU shared service team and now they are investigating the issue. Appreciate your initial glance on the issue.