Object not serializable error on an if then else statement run on spark engine

Noah

I get the following error when I run an if then else error on spark. This runs fine on the local engine. I have if then else statements that run just fine in other sections of my workflow in the DSS. How should I interpret this and how do I debug?

Job failed: Task not serializable, caused by: NotSerializableException: com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue Serialization stack: - object not serializable (class: com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue, value: com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@6990f6af) - writeObject data (class: java.util.ArrayList) - object (class java.util.ArrayList, [com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@6990f6af, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@5080c40e, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@75fb04ee, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@5c863996, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiListConditionValue@1000303a]) - field (class: com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition, name: items, type: interface java.util.List) - object (class com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@14abde3e) - writeObject data (class: java.util.ArrayList) - object (class java.util.ArrayList, [com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@14abde3e, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@6daeaff2, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@59aa646e, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@541c32de, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiCondition@22120799]) - field (class: com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiData, name: conditions, type: interface java.util.List) - object (class com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiData, com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiData@4c33bf70) - field (class: com.dataiku.dip.dataflow.exec.filter.FilterDesc, name: uiData, type: class com.dataiku.dip.dataflow.exec.filter.FilterDesc$FilterUiData) - object (class com.dataiku.dip.dataflow.exec.filter.FilterDesc, com.dataiku.dip.dataflow.exec.filter.FilterDesc@e6d9c25) - field (class: com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc$IfThen, name: filter, type: class com.dataiku.dip.dataflow.exec.filter.FilterDesc) - object (class com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc$IfThen, com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc$IfThen@4eef2522) - writeObject data (class: java.util.ArrayList) - object (class java.util.ArrayList, [com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc$IfThen@4eef2522]) - field (class: com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc, name: elseIfThens, type: interface java.util.List) - object (class com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc, com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc@78181f7f) - field (class: com.dataiku.dip.shaker.processors.transform.visualif.VisualIfRule$Parameter, name: visualIfDesc, type: class com.dataiku.dip.shaker.processors.transform.visualif.VisualIfDesc) - object (class com.dataiku.dip.shaker.processors.transform.visualif.VisualIfRule$Parameter, com.dataiku.dip.shaker.processors.transform.visualif.VisualIfRule$Parameter@509d0d21) - field (class: com.dataiku.dip.shaker.model.ProcessorScriptStep, name: params, type: interface com.dataiku.dip.shaker.model.StepParams) - object (class com.dataiku.dip.shaker.model.ProcessorScriptStep, com.dataiku.dip.shaker.model.ProcessorScriptStep@d6f227f8) - writeObject data (class: scala.collection.immutable.List$SerializationProxy) - object (class scala.collection.immutable.List$SerializationProxy, scala.collection.immutable.List$SerializationProxy@2603a76d) - writeReplace data (class: scala.collection.immutable.List$SerializationProxy) - object (class scala.collection.immutable.$colon$colon, List(com.dataiku.dip.shaker.model.ProcessorScriptStep@d6f227f8, com.dataiku.dip.shaker.model.ProcessorScriptStep@e7334d6e)) - field (class: com.dataiku.dip.shaker.sparkimpl.SimplePipelineCompiler$$anonfun$build$2, name: steps$1, type: class scala.collection.immutable.List) - object (class com.dataiku.dip.shaker.sparkimpl.SimplePipelineCompiler$$anonfun$build$2, <function1>) More info about this error

wfgb

Hello, I am facing the same problem. Did you get an answer?