Drop data set to False deletes the SQL table

Janaat
Janaat Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 30 Partner

Hi,

I create an SQL table dataset inside my custom plugin recipe and try to delete just the dataset without dropping the table in the database. But the table is being dropped whether or not I set drop_data to True or False. Just wanted to check if I'm doing it the right way.

I referred to this documentation

https://developer.dataiku.com/latest/concepts-and-examples/datasets/datasets-other.html#deleting-a-dataset

Thanks

dataset = project.create_sql_table_dataset(table_name, "PostgreSQL", connection, table_name, schema)
dataset_settings = dataset.get_settings()
table_name_with_schema = '"' + schema + '"."' + table_name + '"'
dataset_settings.get_raw_params()["customPostWriteStatements"] = 'ALTER TABLE ' + table_name_with_schema + ' ADD COLUMN id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY; GRANT SELECT , DELETE ON TABLE ' + table_name_with_schema + ' TO psqladmin;'
dataset_settings.save()
src_dataset.copy_to(dataset, sync_schema=True, write_mode='OVERWRITE')
dataset.delete(drop_data=False)

Best Answer

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,399 Dataiker
    Answer ✓

    Hi @jvijayakumar2
    ,
    As discussed on the support ticket your code does not wait for the copy_to to complete thus no table is created at that point.

    You should avoid this by adding ( wait_for_result()

    src_dataset.copy_to(dataset, sync_schema=True, write_mode='OVERWRITE').wait_for_result()


    Thanks

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023, Circle Member Posts: 2,722 Neuron

    The API reference says the drop_data=False but you have drop_na=False in your code.

  • Janaat
    Janaat Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 30 Partner

    It was drop_data=False that I was using. Corrected the sample snippet.

  • Janaat
    Janaat Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 30 Partner

    Thanks @AlexT
    I was able to get it working by adding wait_for_result()

Setup Info
    Tags
      Help me…