How to use custom DataSet plugin for data deletion
Hi,
I have a custom dataset plugin to read data from a data store. I would like to know what is the best way to use the same custom plugin to support data deletion.
I am re-using the existing custom plugin user interface to allow user input for for deletion. I am using the generate_rows method in Connector class for delete operation. But I am not happy with this approach.
Would anyone suggest a better way to support data deletion in a custom dataset plugin.
Thanks, and below is my pseudo code:
def generate_rows(self, dataset_schema=None, dataset_partitioning=None,
partition_id=None, records_limit = -1):
"""
The main reading method.
Returns a generator over the rows of the dataset (or partition)
Each yielded row must be a dictionary, indexed by column name.
The dataset schema and partitioning are given for information purpose.
operation_type = self.config.get('operation_type', '')
if operation_type == 'read':
read_data(records_limit):
elif operation_type == 'delete':
delete_data()
return None
else:
raise ValueError('Invalid operation type.')