Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
I have a custom dataset plugin to read data from a data store. I would like to know what is the best way to use the same custom plugin to support data deletion.
I am re-using the existing custom plugin user interface to allow user input for for deletion. I am using the generate_rows method in Connector class for delete operation. But I am not happy with this approach.
Would anyone suggest a better way to support data deletion in a custom dataset plugin.
Thanks, and below is my pseudo code:
def generate_rows(self, dataset_schema=None, dataset_partitioning=None,
partition_id=None, records_limit = -1):
"""
The main reading method.
Returns a generator over the rows of the dataset (or partition)
Each yielded row must be a dictionary, indexed by column name.
The dataset schema and partitioning are given for information purpose.
operation_type = self.config.get('operation_type', '')
if operation_type == 'read':
read_data(records_limit):
elif operation_type == 'delete':
delete_data()
return None
else:
raise ValueError('Invalid operation type.')