Partial data set clear in scenario steps

Options
Ankur5289
Ankur5289 Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered Posts: 27 Partner

i have to delete data in a data set based on some conditions.

I want to achieve this via a auto trigger scenario maybe once in a day.

Do you think this will be able to achieve in dataiku? Pls let me know

Answers

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker
    Options

    Hello,

    While there is no scenario step that can clear data based on specific conditions explicitly, this might be possible depending on the type of conditions / consistency (stays the same) of the conditions.

    For example, if there was a dataset in which everyday you want to remove rows that contain 'XYZ', you could run a series of steps that:

    1. filters the existing dataset using a filter recipe, (build dataset step)
    2. clears the initial existing dataset, (clear dataset step)
    3. writes the output of filtered recipe back to initial recipe (python step)

    Alternatively, you could do all of this in a python step, as well.

    Please let me know if you have any questions.

    Roy

Setup Info
    Tags
      Help me…