Interact with partition in a exporter plugin

Jordan
Jordan Registered Posts: 8 ✭✭✭✭

If my dataset is partitioned how can i interact with partitions in a exporter plugin ? example if i want to export all partitions or only one how can i do ? is it in the plugin or in dataset

Best Answer

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Answer ✓

    Hi,

    Your exporter is not aware of partitions. In fact, your exporter is not even aware that it's exporting a dataset. The reason is that when you write an exporter plugin, it can actually be used for exporting lots of different things in DSS:

    • A dataset
    • Processed data from a preparation script in a visual analysis
    • Predicted data from a visual ML analysis
    • Various "tables of data" here and there in the UI (for example, coefficients for a regression, schema for a dataset)
    • SQL notebook query results
    • Pandas dataframe from a Jupyter notebook

    So an exporter plugin just receives a stream of data without knowing their provenance.

    However, when you export a partitioned dataset, you'll have the option, when clicking on the "Export" button to select which partitions you want to export. DSS will then handle the work of selecting the partitions that will be sent to your exporter plugin

Answers

Setup Info
    Tags
      Help me…