Partial data set clear in scenario steps
i have to delete data in a data set based on some conditions.
I want to achieve this via a auto trigger scenario maybe once in a day.
Do you think this will be able to achieve in dataiku? Pls let me know
Answers
-
RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 DataikerHello,
While there is no scenario step that can clear data based on specific conditions explicitly, this might be possible depending on the type of conditions / consistency (stays the same) of the conditions.
For example, if there was a dataset in which everyday you want to remove rows that contain 'XYZ', you could run a series of steps that:
1. filters the existing dataset using a filter recipe, (build dataset step)
2. clears the initial existing dataset, (clear dataset step)
3. writes the output of filtered recipe back to initial recipe (python step)Alternatively, you could do all of this in a python step, as well.
Please let me know if you have any questions.
Roy
-
can you explain the 3rd steps ? when I am writing python receipe I need to select the existing dataset , so that I can write back to same old dataset a, after we clear the data from it. but when you try to add this as output with python receipe it simply doesn't allow you to add it .
is there workaround for it ?
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,876 NeuronPlease post a new thread with your question.
