InvocationTargetException while writing dataset into postgreSql DB.

srikanth_ch
srikanth_ch Registered Posts: 3

Hi Team,

I am new to DataIKU and when I try to execute one of the python recipe which stores dataframe into Postgresql table it throws exception .

Exception trace says, duplicate column, but the table is empty.

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "pk_markit_rejections"
Detail: Key (business_area, commodity_name, validation_date, team_name, trading_hub_full_name, trading_hub, ac_symbol, term_type, term_year, term_start_date)=(GPTA, Power, 2023-05-08, East, ZZZZZZ - 5x16 / 2x16 / 7x8, YYYYYYY - 5x16 / 2x16 / 7x8 - Off, XXXXXXXX, Month, 2023, 2023-09-01) already exists.
Where: COPY markit_rejections, line 2164


Operating system used: Windows 11

Tagged:

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,215 Dataiker

    Hi @srikanth_ch
    ,

    It looks like you are trying to write to an existing table or table created with a custom SQL creation script?
    As typically, this wouldn't come up when using a managed SQL dataset in DSS.

    To resolve this issue, you could either modify the data that is being inserted to ensure that the primary key values are unique or remove the primary key constraint to allow for duplicates.

    Thanks

  • srikanth_ch
    srikanth_ch Registered Posts: 3

    Hi @AlexT

    Thank you for the response. Yes, I am trying to write into existing table, but the record which I am inserting is not present in the table. The insertion record is unique.

    xyz_dataset= dataiku.Dataset("xyz")
    xyz_dataset.write_with_schema(abc_df)

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,215 Dataiker

    The error would suggest the record is already present so hard to explain why you would be seeing this. I would suggest you open a support ticket with job diagnostics so we can investigate further.

    Have you tried writing to a new dataset/table managed by DSS are you seeing any issues. When writing from DSS its expected you use managed dataset and not existing tables, existing tables should be used as input datasets.


  • srikanth_ch
    srikanth_ch Registered Posts: 3

    Thanks @AlexT

    I raised this concern to internal dataIKU shared service team and now they are investigating the issue. Appreciate your initial glance on the issue.

Setup Info
    Tags
      Help me…