InvocationTargetException while writing dataset into postgreSql DB.
Hi Team,
I am new to DataIKU and when I try to execute one of the python recipe which stores dataframe into Postgresql table it throws exception .
Exception trace says, duplicate column, but the table is empty.
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_markit_rejections"
Detail: Key (business_area, commodity_name, validation_date, team_name, trading_hub_full_name, trading_hub, ac_symbol, term_type, term_year, term_start_date)=(GPTA, Power, 2023-05-08, East, ZZZZZZ - 5x16 / 2x16 / 7x8, YYYYYYY - 5x16 / 2x16 / 7x8 - Off, XXXXXXXX, Month, 2023, 2023-09-01) already exists.
Where: COPY markit_rejections, line 2164
Operating system used: Windows 11
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 DataikerOptionsHi @srikanth_ch
,It looks like you are trying to write to an existing table or table created with a custom SQL creation script?
As typically, this wouldn't come up when using a managed SQL dataset in DSS.
To resolve this issue, you could either modify the data that is being inserted to ensure that the primary key values are unique or remove the primary key constraint to allow for duplicates.
Thanks -
Hi @AlexT
Thank you for the response. Yes, I am trying to write into existing table, but the record which I am inserting is not present in the table. The insertion record is unique.
xyz_dataset= dataiku.Dataset("xyz")
xyz_dataset.write_with_schema(abc_df) -
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 DataikerOptions
The error would suggest the record is already present so hard to explain why you would be seeing this. I would suggest you open a support ticket with job diagnostics so we can investigate further.
Have you tried writing to a new dataset/table managed by DSS are you seeing any issues. When writing from DSS its expected you use managed dataset and not existing tables, existing tables should be used as input datasets. -
Thanks @AlexT
I raised this concern to internal dataIKU shared service team and now they are investigating the issue. Appreciate your initial glance on the issue.