Scoring fails.

Options
mattmagic
mattmagic Registered Posts: 12 ✭✭✭✭

Hi,

from time to time the scoring of a prediction model fails.

The only thing that seems to solve it is to train a new model and deploy a completely different algorithm.



Here is my log:


[09:23:38] [INFO] [dku.utils] - 2017-09-19 09:23:38,188 INFO Reading with dtypes: {u'Domain': None, u'DescriptionProcessed': 'str'}<BR />[09:23:38] [INFO] [dku.utils] - 2017-09-19 09:23:38,188 INFO Column 0 = Domain (dtype=None)<BR />[09:23:38] [INFO] [dku.utils] - 2017-09-19 09:23:38,188 INFO Column 1 = Source (dtype=None)<BR />[09:23:38] [INFO] [dku.utils] - 2017-09-19 09:23:38,188 INFO Column 2 = DescriptionProcessed (dtype=str)<BR />[09:23:38] [INFO] [dku.utils] - 2017-09-19 09:23:38,283 INFO Starting dataframes iterator<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,688 INFO Got a dataframe : (100000, 3)<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,688 INFO Coercion done<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,688 INFO NORMALIZED: Domain -> object<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,688 INFO NORMALIZED: Source -> object<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,688 INFO NORMALIZED: DescriptionProcessed -> object<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,689 INFO Processing it<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,697 INFO Set MF index len 100000<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,697 DEBUG PROCESS WITH Step:MultipleImputeMissingFromInput<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,697 DEBUG MIMIFI: Imputing with map {}<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,698 DEBUG PROCESS WITH Step:FlushDFBuilder(num_flagonly)<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,698 DEBUG PROCESS WITH Step:MultipleImputeMissingFromInput<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,698 DEBUG MIMIFI: Imputing with map {}<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,698 DEBUG PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)<BR />[09:23:41] [INFO] [dku.utils] - 2017-09-19 09:23:41,698 DEBUG PROCESS WITH <class 'dataiku.doctor.preprocessing.dataframe_preprocessing.TextTFIDFVectorizerProcessor'> (DescriptionProcessed)<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 DEBUG PROCESS WITH Step:FlushDFBuilder(interaction)<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 DEBUG PROCESS WITH Step:DumpPipelineState<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 INFO ********* Pipieline state (Before feature selection)<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 INFO input_df= (100000, 3) <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 INFO current_mf=(100000, 26558) <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 INFO PPR: <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,616 DEBUG PROCESS WITH Step:EmitCurrentMFAsResult<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO Set MF index len 100000<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 DEBUG PROCESS WITH Step:DumpPipelineState<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO ********* Pipieline state (At end)<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO input_df= (100000, 3) <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO current_mf=(0, 0) <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO PPR: <BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((100000, 26558))<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO Predicting it<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,625 INFO Prepare to predict ...<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Init write session: tcenfKZdgi<BR />[09:24:02] [DEBUG] [dku.jobs] - Command /tintercom/datasets/init-write-session processed in 14ms<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,679 INFO Initializing write data stream (tcenfKZdgi)<BR />[09:24:02] [INFO] [dku.jobs] - Connects using API ticket<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,682 INFO Waiting for data to send ...<BR />[09:24:02] [DEBUG] [dku.jobs] - Received command : /tintercom/datasets/wait-write-session<BR />[09:24:02] [INFO] [dku.utils] - 2017-09-19 09:24:02,682 INFO Got end mark, ending send<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Creating output writer<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Initializing output writer<BR />[09:24:02] [INFO] [dku.connections.sql.provider] - Connecting to jdbc:postgresql://dataiku.ct2brvwy8za8.us-east-1.rds.amazonaws.com:5432/Dataiku with props: {}<BR />[09:24:02] [DEBUG] [dku.connections.sql.provider] - Driver version 9.0<BR />[09:24:02] [INFO] [dku.connections.sql.provider] - Driver PostgreSQL Native Driver (JDBC 4.0) PostgreSQL 9.0 JDBC4 (build 801) (9.0)<BR />[09:24:02] [INFO] [dku.connections.sql.provider] - Database PostgreSQL 9.5.4 (9.5) rowSize=1073741824 stmts=0<BR />[09:24:02] [DEBUG] [dku.connections.sql.provider] - Set autocommit=false on conn=Dataiku_DB<BR />[09:24:02] [INFO] [dku.sql.generic] - Dropping table<BR />[09:24:02] [INFO] [dku.dataset.sql] - Executing statement:<BR />[09:24:02] [INFO] [dku.dataset.sql] - DROP TABLE "CLUSTERREPORTCLUSTERINGNEW_cb_descriptions_consumerelectronics_classified"<BR />[09:24:02] [INFO] [dku.dataset.sql] - Statement done<BR />[09:24:02] [INFO] [dku.sql.generic] - Creating table<BR />[09:24:02] [INFO] [dku.dataset.sql] - Executing statement:<BR />[09:24:02] [INFO] [dku.dataset.sql] - CREATE TABLE "CLUSTERREPORTCLUSTERINGNEW_cb_descriptions_consumerelectronics_classified" (<BR /> "Domain" text,<BR /> "Source" text,<BR /> "DescriptionProcessed" text,<BR /> "proba_0.0" double precision,<BR /> "proba_1.0" double precision,<BR /> "prediction" double precision<BR />)<BR />[09:24:02] [INFO] [dku.dataset.sql] - Statement done<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Done initializing output writer<BR />[09:24:02] [INFO] [dku.output.sql.pglike] - Copy done, copied 0 records<BR />[09:24:02] [INFO] [dku.connections.sql.provider] - Commit conn=Dataiku_DB<BR />[09:24:02] [DEBUG] [dku.connections.sql.provider] - Close conn=Dataiku_DB<BR />[09:24:02] [INFO] [dku.output.sql.pglike] - Transaction done, copied 0 records<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Pushed data to write session tcenfKZdgi : 0 rows<BR />[09:24:02] [DEBUG] [dku.jobs] - Command /tintercom/datasets/push-data processed in 144ms<BR />[09:24:02] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Finished write session: tcenfKZdgi<BR />[09:24:02] [DEBUG] [dku.jobs] - Command /tintercom/datasets/wait-write-session processed in 146ms<BR />[09:24:02] [INFO] [dku.utils] - 0 rows successfully written (tcenfKZdgi)<BR />[09:24:02] [INFO] [dku.utils] - Traceback (most recent call last):<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dss/condaenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main<BR />[09:24:02] [INFO] [dku.utils] - "__main__", fname, loader, pkg_name)<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dss/condaenv/lib/python2.7/runpy.py", line 72, in _run_code<BR />[09:24:02] [INFO] [dku.utils] - exec code in run_globals<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 146, in <module><BR />[09:24:02] [INFO] [dku.utils] - json.load_from_filepath(sys.argv[7]))<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 133, in main<BR />[09:24:02] [INFO] [dku.utils] - for output_df in output_generator():<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 78, in output_generator<BR />[09:24:02] [INFO] [dku.utils] - output_probas=recipe_desc["outputProbabilities"])<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/prediction/classification_scoring.py", line 206, in binary_classification_predict<BR />[09:24:02] [INFO] [dku.utils] - (pred_df, proba_df) = binary_classification_predict_ex(clf, modeling_params, target_map, threshold, transformed, output_probas)<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/prediction/classification_scoring.py", line 157, in binary_classification_predict_ex<BR />[09:24:02] [INFO] [dku.utils] - features_X_df = features_X.as_dataframe()<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dataiku-dss-4.0.8/python/dataiku/doctor/multiframe.py", line 269, in as_dataframe<BR />[09:24:02] [INFO] [dku.utils] - blkdf = pd.DataFrame(blk.matrix.toarray(), columns=blk.names)<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dss/condaenv/lib/python2.7/site-packages/scipy/sparse/compressed.py", line 920, in toarray<BR />[09:24:02] [INFO] [dku.utils] - return self.tocoo(copy=False).toarray(order=order, out=out)<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dss/condaenv/lib/python2.7/site-packages/scipy/sparse/coo.py", line 252, in toarray<BR />[09:24:02] [INFO] [dku.utils] - B = self._process_toarray_args(order, out)<BR />[09:24:02] [INFO] [dku.utils] - File "/home/dataiku/dss/condaenv/lib/python2.7/site-packages/scipy/sparse/base.py", line 1009, in _process_toarray_args<BR />[09:24:02] [INFO] [dku.utils] - return np.zeros(self.shape, dtype=self.dtype, order=order)<BR />[09:24:02] [INFO] [dku.utils] - MemoryError<BR />[09:24:02] [INFO] [dku.flow.activity] - Run thread failed for activity score_CB_descriptions_final_without_labels_14_NP<BR />com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 1). More info might be available in the logs.<BR /> at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:311)<BR /> at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:231)<BR /> at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeModule(AbstractPythonRecipeRunner.java:47)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runOriginalPython(PredictionScoringRecipeRunner.java:388)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runWithOriginalEngine(PredictionScoringRecipeRunner.java:294)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.run(PredictionScoringRecipeRunner.java:220)<BR /> at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:353)<BR />[09:24:03] [INFO] [dku.flow.activity] running score_CB_descriptions_final_without_labels_14_NP - activity is finished<BR />[09:24:03] [ERROR] [dku.flow.activity] running score_CB_descriptions_final_without_labels_14_NP - Activity failed<BR />com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 1). More info might be available in the logs.<BR /> at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:311)<BR /> at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:231)<BR /> at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeModule(AbstractPythonRecipeRunner.java:47)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runOriginalPython(PredictionScoringRecipeRunner.java:388)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runWithOriginalEngine(PredictionScoringRecipeRunner.java:294)<BR /> at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.run(PredictionScoringRecipeRunner.java:220)<BR /> at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:353)



I have wasted hours to create and re-create models.

Does anyone have a similar experience?

Best Answer

  • cperdigou
    cperdigou Alpha Tester, Dataiker Alumni Posts: 115 ✭✭✭✭✭✭✭
    Answer ✓
    Options
    Hello,

    Your input data is very large, with 26k+ columns. Scoring is done on chunks of 100k rows (this cannot be changed), meaning that in your case this creates a 100k * 26k matrix, which does not fit in memory and leads to the "Memory error" message that you have in your logs.

Answers

Setup Info
    Tags
      Help me…