Filter recipe : How to avoid stop processing when there are no matched records
databirds11
Registered Posts: 2 ✭
Hi dataiku users,
I want to know how to resolve the situation in subject.
I use filter recipe only for processing the exception data, and stack with main data after that, so if there are no filtered records in output dataset, no problem.
but in dataiku, if there are not all data sets in stack recipe, return error and stop processing in that time.
Do you know how to ?
Best Answer
-
I could avoid error by changing engine of filter recipe, "Spark" to "DSS".
It seems DSS engine allow to create 0 record dataset, but Spark engine is not.
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,054 Neuron
Have a read at this post which basically does exactly what you want which is to conditionally control the execution of the scenario steps based on some data:
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,054 Neuron
If you are using Databricks this is a bug. Raise it with Dataiku Support.