[2021/03/31-14:37:12.803] [qtp2128169374-25] [INFO] [dku.dataset.sql] - Statement done [2021/03/31-14:37:12.803] [qtp2128169374-25] [DEBUG] [dku.connections.sql.provider] - Commit conn=DW_OLAPRW-05JTUKp [2021/03/31-14:37:12.960] [qtp2128169374-25] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Done initializing output writer [2021/03/31-14:37:12.962] [qtp2128169374-25] [INFO] [dku.output.sql] - committing, appended 0 failedRows=0 rowsWithFailedCells=0 [2021/03/31-14:37:12.963] [qtp2128169374-25] [DEBUG] [dku.connections.sql.provider] - Commit conn=DW_OLAPRW-05JTUKp [2021/03/31-14:37:13.004] [qtp2128169374-25] [DEBUG] [dku.connections.sql.provider] - Close conn=DW_OLAPRW-05JTUKp [2021/03/31-14:37:13.007] [qtp2128169374-25] [INFO] [dku.resourceusage] - Reporting completion of CRU:{"type":"SQL_CONNECTION","id":"2eHcNeTxU0knJJTR","startTime":1617215832630,"sqlConnection":{"connection":"DW_OLAPRW"}} [2021/03/31-14:37:13.008] [qtp2128169374-25] [INFO] [dku.usage.computeresource.jek] - Reporting completion of resource usage: {"type":"SQL_CONNECTION","id":"2eHcNeTxU0knJJTR","startTime":1617215832630,"endTime":1617215833007,"sqlConnection":{"connection":"DW_OLAPRW"}} [2021/03/31-14:37:13.008] [qtp2128169374-25] [INFO] [dku.output.sql] - Transaction done [2021/03/31-14:37:13.008] [qtp2128169374-25] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Pushed data to write session qCMUozr9eC : 0 rows [2021/03/31-14:37:13.009] [qtp2128169374-25] [DEBUG] [dku.jobs] - Command /tintercom/datasets/push-data processed in 962ms [2021/03/31-14:37:13.008] [qtp2128169374-21] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Finished write session: qCMUozr9eC [2021/03/31-14:37:13.011] [qtp2128169374-21] [DEBUG] [dku.jobs] - Command /tintercom/datasets/wait-write-session processed in 964ms [2021/03/31-14:37:13.014] [null-out-41] [INFO] [dku.utils] - 0 rows successfully written (qCMUozr9eC) [2021/03/31-14:37:13.016] [null-err-42] [INFO] [dku.utils] - Traceback (most recent call last): [2021/03/31-14:37:13.017] [null-err-42] [INFO] [dku.utils] - File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main [2021/03/31-14:37:13.020] [null-err-42] [INFO] [dku.utils] - "__main__", fname, loader, pkg_name) [2021/03/31-14:37:13.020] [null-err-42] [INFO] [dku.utils] - File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code [2021/03/31-14:37:13.021] [null-err-42] [INFO] [dku.utils] - exec code in run_globals [2021/03/31-14:37:13.021] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 275, in [2021/03/31-14:37:13.023] [null-err-42] [INFO] [dku.utils] - dkujson.load_from_filepath(sys.argv[7])) [2021/03/31-14:37:13.024] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 257, in main [2021/03/31-14:37:13.024] [null-err-42] [INFO] [dku.utils] - for output_df in output_generator(): [2021/03/31-14:37:13.024] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/prediction/reg_scoring_recipe.py", line 242, in output_generator [2021/03/31-14:37:13.025] [null-err-42] [INFO] [dku.utils] - shapley_background_size=shapley_background_size) [2021/03/31-14:37:13.025] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/individual_explainer.py", line 168, in explain [2021/03/31-14:37:13.025] [null-err-42] [INFO] [dku.utils] - columns_to_compute, progress) [2021/03/31-14:37:13.026] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/individual_explainer.py", line 300, in compute_ice_values [2021/03/31-14:37:13.026] [null-err-42] [INFO] [dku.utils] - scores_arr = self._get_prediction_results(frankenstein_df).scores_to_explain_df.values [2021/03/31-14:37:13.026] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/individual_explainer.py", line 487, in _get_prediction_results [2021/03/31-14:37:13.027] [null-err-42] [INFO] [dku.utils] - transformed_a, predicted_df = self._predict_and_get_transformed_df(observations) [2021/03/31-14:37:13.027] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/individual_explainer.py", line 506, in _predict_and_get_transformed_df [2021/03/31-14:37:13.027] [null-err-42] [INFO] [dku.utils] - transformed_a, input_index, _ = self._predictor.preprocessing.preprocess(observations) [2021/03/31-14:37:13.027] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/core/saved_model.py", line 416, in preprocess [2021/03/31-14:37:13.028] [null-err-42] [INFO] [dku.utils] - X = prepare_multiframe(transformed_train, self.modeling_params)[0] [2021/03/31-14:37:13.028] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/prediction/common.py", line 728, in prepare_multiframe [2021/03/31-14:37:13.029] [null-err-42] [INFO] [dku.utils] - return train_X.as_np_array(), False [2021/03/31-14:37:13.030] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/multiframe.py", line 266, in as_np_array [2021/03/31-14:37:13.030] [null-err-42] [INFO] [dku.utils] - val = MultiFrame.block_as_np_array(blk) [2021/03/31-14:37:13.031] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python/dataiku/doctor/multiframe.py", line 276, in block_as_np_array [2021/03/31-14:37:13.031] [null-err-42] [INFO] [dku.utils] - val = blk.matrix.toarray() [2021/03/31-14:37:13.031] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python.packages/scipy/sparse/compressed.py", line 947, in toarray [2021/03/31-14:37:13.033] [null-err-42] [INFO] [dku.utils] - out = self._process_toarray_args(order, out) [2021/03/31-14:37:13.034] [null-err-42] [INFO] [dku.utils] - File "/opt/dataiku/install_dir/dataiku-dss-8.0.2/python.packages/scipy/sparse/base.py", line 1184, in _process_toarray_args [2021/03/31-14:37:13.035] [null-err-42] [INFO] [dku.utils] - return np.zeros(self.shape, dtype=self.dtype, order=order) [2021/03/31-14:37:13.036] [null-err-42] [INFO] [dku.utils] - MemoryError [2021/03/31-14:37:14.831] [FRT-35-FlowRunnable] [WARN] [dku.resource] - stat file for pid 71956 does not exist. Process died? [2021/03/31-14:37:14.832] [FRT-35-FlowRunnable] [INFO] [dku.resourceusage] - Reporting completion of CRU:{"context":{"type":"JOB_ACTIVITY","authIdentifier":"212435620","projectKey":"PREJOBSAFETYRATINGS","jobId":"Build_PrejobEventProbability_2021-03-31T18-36-30.888","activityId":"score_JobEventOccured_joined_joined_prepared_joined_joined_NP","activityType":"recipe","recipeType":"prediction_scoring","recipeName":"score_JobEventOccured_joined_joined_prepared_joined_joined"},"type":"LOCAL_PROCESS","id":"85gJ900yejBwEmXb","startTime":1617215792632,"localProcess":{"pid":71956,"commandName":"/opt/dataiku/data_dir/bin/python","cpuUserTimeMS":38480,"cpuSystemTimeMS":9640,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":10,"cpuTotalMS":48130,"cpuCurrent":0.8658268346330734,"vmSizeMB":5589,"vmRSSMB":4021,"vmHWMMB":4021,"vmRSSAnonMB":4002,"vmDataMB":4924,"vmSizePeakMB":5589,"vmRSSPeakMB":4021,"vmRSSTotalMBS":65090,"majorFaults":0,"childrenMajorFaults":0}} [2021/03/31-14:37:14.832] [FRT-35-FlowRunnable] [INFO] [dku.usage.computeresource.jek] - Reporting completion of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"212435620","projectKey":"PREJOBSAFETYRATINGS","jobId":"Build_PrejobEventProbability_2021-03-31T18-36-30.888","activityId":"score_JobEventOccured_joined_joined_prepared_joined_joined_NP","activityType":"recipe","recipeType":"prediction_scoring","recipeName":"score_JobEventOccured_joined_joined_prepared_joined_joined"},"type":"LOCAL_PROCESS","id":"85gJ900yejBwEmXb","startTime":1617215792632,"endTime":1617215834832,"localProcess":{"pid":71956,"commandName":"/opt/dataiku/data_dir/bin/python","cpuUserTimeMS":38480,"cpuSystemTimeMS":9640,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":10,"cpuTotalMS":48130,"cpuCurrent":0.8658268346330734,"vmSizeMB":5589,"vmRSSMB":4021,"vmHWMMB":4021,"vmRSSAnonMB":4002,"vmDataMB":4924,"vmSizePeakMB":5589,"vmRSSPeakMB":4021,"vmRSSTotalMBS":65090,"majorFaults":0,"childrenMajorFaults":0}} [2021/03/31-14:37:14.850] [FRT-35-FlowRunnable] [INFO] [dku.flow.activity] - Run thread failed for activity score_JobEventOccured_joined_joined_prepared_joined_joined_NP com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 1). More info might be available in the logs. at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103) at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeModule(AbstractPythonRecipeRunner.java:64) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner$3.run(PredictionScoringRecipeRunner.java:589) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runPython(PredictionScoringRecipeRunner.java:627) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runWithOriginalEngine(PredictionScoringRecipeRunner.java:384) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.run(PredictionScoringRecipeRunner.java:259) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374) [2021/03/31-14:37:14.998] [ActivityExecutor-30] [INFO] [dku.flow.activity] running score_JobEventOccured_joined_joined_prepared_joined_joined_NP - activity is finished [2021/03/31-14:37:15.000] [ActivityExecutor-30] [ERROR] [dku.flow.activity] running score_JobEventOccured_joined_joined_prepared_joined_joined_NP - Activity failed com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 1). More info might be available in the logs. at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103) at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeModule(AbstractPythonRecipeRunner.java:64) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner$3.run(PredictionScoringRecipeRunner.java:589) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runPython(PredictionScoringRecipeRunner.java:627) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.runWithOriginalEngine(PredictionScoringRecipeRunner.java:384) at com.dataiku.dip.analysis.ml.prediction.flow.PredictionScoringRecipeRunner.run(PredictionScoringRecipeRunner.java:259) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374) [2021/03/31-14:37:15.000] [ActivityExecutor-30] [INFO] [dku.flow.activity] running score_JobEventOccured_joined_joined_prepared_joined_joined_NP - Executing default post-activity lifecycle hook [2021/03/31-14:37:15.004] [ActivityExecutor-30] [INFO] [dku.flow.activity] running score_JobEventOccured_joined_joined_prepared_joined_joined_NP - Removing samples for PREJOBSAFETYRATINGS.PrejobEventProbability [2021/03/31-14:37:15.031] [ActivityExecutor-30] [INFO] [dku.flow.activity] running score_JobEventOccured_joined_joined_prepared_joined_joined_NP - Done post-activity tasks [2021/03/31-14:37:15.547] [qtp2128169374-21] [INFO] [dku.jobs] - Connects using Shared secret [2021/03/31-14:37:15.547] [qtp2128169374-21] [DEBUG] [dku.jobs] - Received command : /pintercom/stop_session [2021/03/31-14:37:15.550] [qtp2128169374-21] [DEBUG] [dku.jobs] - Command /pintercom/stop_session processed in 3ms