Interact with partition in a exporter plugin
If my dataset is partitioned how can i interact with partitions in a exporter plugin ? example if i want to export all partitions or only one how can i do ? is it in the plugin or in dataset
Best Answer
-
Hi,
Your exporter is not aware of partitions. In fact, your exporter is not even aware that it's exporting a dataset. The reason is that when you write an exporter plugin, it can actually be used for exporting lots of different things in DSS:
- A dataset
- Processed data from a preparation script in a visual analysis
- Predicted data from a visual ML analysis
- Various "tables of data" here and there in the UI (for example, coefficients for a regression, schema for a dataset)
- SQL notebook query results
- Pandas dataframe from a Jupyter notebook
So an exporter plugin just receives a stream of data without knowing their provenance.
However, when you export a partitioned dataset, you'll have the option, when clicking on the "Export" button to select which partitions you want to export. DSS will then handle the work of selecting the partitions that will be sent to your exporter plugin
Answers
-
Thanks you for the answer
So there is no way i can retrieve the value of the partition that i choose , to call it in the exporter plugin code so that i can pass to an api to create folder with the value
-
This is not possible indeed. You may want to write a recipe instead of an exporter for that need.
-
thanks you