Partial data set clear in scenario steps

Ankur5289
Level 3
Partial data set clear in scenario steps

i have to delete data in a data set based on some conditions.

I want to achieve this via a auto trigger scenario maybe once in a day.

Do you think this will be able to achieve in dataiku? Pls let me know  

0 Kudos
1 Reply
RoyE
Dataiker

Hello,

While there is no scenario step that can clear data based on specific conditions explicitly, this might be possible depending on the type of conditions / consistency (stays the same) of the conditions.

For example, if there was a dataset in which everyday you want to remove rows that contain 'XYZ', you could run a series of steps that:

1. filters the existing dataset using a filter recipe, (build dataset step)
2. clears the initial existing dataset, (clear dataset step)
3. writes the output of filtered recipe back to initial recipe (python step)

Alternatively, you could do all of this in a python step, as well. 

Please let me know if you have any questions.

Roy

0 Kudos