Partial data set clear in scenario steps

Ankur5289
Ankur5289 Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered Posts: 27 Partner

i have to delete data in a data set based on some conditions.

I want to achieve this via a auto trigger scenario maybe once in a day.

Do you think this will be able to achieve in dataiku? Pls let me know

Answers

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker

    Hello,

    While there is no scenario step that can clear data based on specific conditions explicitly, this might be possible depending on the type of conditions / consistency (stays the same) of the conditions.

    For example, if there was a dataset in which everyday you want to remove rows that contain 'XYZ', you could run a series of steps that:

    1. filters the existing dataset using a filter recipe, (build dataset step)
    2. clears the initial existing dataset, (clear dataset step)
    3. writes the output of filtered recipe back to initial recipe (python step)

    Alternatively, you could do all of this in a python step, as well.

    Please let me know if you have any questions.

    Roy

  • Chakshu
    Chakshu Registered Posts: 1 ✭✭

    can you explain the 3rd steps ? when I am writing python receipe I need to select the existing dataset , so that I can write back to same old dataset a, after we clear the data from it. but when you try to add this as output with python receipe it simply doesn't allow you to add it .

    is there workaround for it ?

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,124 Neuron
Setup Info
    Tags
      Help me…