InvocationTargetException while writing dataset into postgreSql DB.

srikanth_ch
Level 1
InvocationTargetException while writing dataset into postgreSql DB.

Hi Team,

I am new to DataIKU and when I try to execute one of the python recipe which stores dataframe into Postgresql table it throws exception . 

Exception trace says, duplicate column, but the table is empty.

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_markit_rejections"
Detail: Key (business_area, commodity_name, validation_date, team_name, trading_hub_full_name, trading_hub, ac_symbol, term_type, term_year, term_start_date)=(GPTA, Power, 2023-05-08, East, ZZZZZZ - 5x16 / 2x16 / 7x8, YYYYYYY - 5x16 / 2x16 / 7x8 - Off, XXXXXXXX, Month, 2023, 2023-09-01) already exists.
Where: COPY markit_rejections, line 2164

 


Operating system used: Windows 11

 

0 Kudos
4 Replies
AlexT
Dataiker

Hi @srikanth_ch ,

It looks like you are trying to write to an existing table or table created with a custom SQL creation script?
As typically, this wouldn't come up when using a managed SQL dataset in DSS.

To resolve this issue, you could either modify the data that is being inserted to ensure that the primary key values are unique or remove the primary key constraint to allow for duplicates.

Thanks

0 Kudos
srikanth_ch
Level 1
Author

Hi @AlexT 

Thank you for the response. Yes, I am trying to write into existing table, but the record which I am inserting is not present in the table. The insertion record is unique.

xyz_dataset= dataiku.Dataset("xyz")
xyz_dataset.write_with_schema(abc_df)

0 Kudos
AlexT
Dataiker

The error would suggest the record is already present so hard to explain why you would be seeing this. I would suggest you open a support ticket with job diagnostics so we can investigate further. 

Have you tried writing to a new dataset/table managed by DSS are you seeing any issues. When writing from DSS its expected you use managed dataset and not existing tables, existing tables should be used as input datasets. 


0 Kudos
srikanth_ch
Level 1
Author

Thanks @AlexT 

I raised this concern to internal dataIKU shared service team and now they are investigating the issue. Appreciate your initial glance on the issue. 

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku