TypeError: copy() got an unexpected keyword argument 'deep'

Sup

currently we have 2 prediction model and we setup the prediction model in the next recipe

# setup efficiency function
efficiency = lambda b: pred_1.predict(np.array([b]))[0] /pred_2.predict(np.array([b]))[0]

# run optimization algorithm

result = optimize.minimize(efficiency,x_init.iloc[0], constraints=constraints, method='COBYLA')

with error msg

2022/12/16-02:21:52.610] [null-err-40] [INFO] [dku.utils]  -   File "/opt/dataiku-dss-11.0.2/python/dataiku/core/saved_model.py", line 232, in _renormalize_dates
[2022/12/16-02:21:52.610] [null-err-40] [INFO] [dku.utils]  -     df = df.copy(deep=False)
[2022/12/16-02:21:52.610] [null-err-40] [INFO] [dku.utils]  - TypeError: copy() got an unexpected keyword argument 'deep'
[2022/12/16-02:21:52.611] [null-err-40] [INFO] [dku.utils]  - End Python stack
[2022/12/16-02:21:52.611] [null-err-40] [INFO] [dku.utils]  - 2022-12-16 02:21:52,606 INFO Check if spark is available
[2022/12/16-02:21:52.611] [null-err-40] [INFO] [dku.utils]  - 2022-12-16 02:21:52,607 INFO Not stopping a spark context: No module named 'pyspark'
[2022/12/16-02:21:52.839] [FRT-37-FlowRunnable] [WARN] [dku.resource] act.compute_NL9Ayu9Z_NP - stat file for pid 4623 does not exist. Process died?
[2022/12/16-02:21:52.840] [FRT-37-FlowRunnable] [DEBUG] [dku.resourceusage] act.compute_NL9Ayu9Z_NP - Reporting completion of CRU:{"context":{"type":"JOB_ACTIVITY","authIdentifier":"sucipto_hardinata","projectKey":"SPOPTIMIZATION","jobId":"Build_projection_output__NP__2022-12-16T02-21-49.030","activityId":"compute_NL9Ayu9Z_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_NL9Ayu9Z"},"type":"LOCAL_PROCESS","id":"pt38VccsgtPeZhdd","startTime":1671157310112,"localProcess":{"pid":4623,"commandName":"/opt/DATA_DIR/code-envs/python/Python37/bin/python","cpuUserTimeMS":20,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":20,"cpuCurrent":0.0,"vmSizeMB":11,"vmRSSMB":7,"vmHWMMB":7,"vmRSSAnonMB":3,"vmDataMB":3,"vmSizePeakMB":11,"vmRSSPeakMB":7,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}}
[2022/12/16-02:21:52.841] [FRT-37-FlowRunnable] [INFO] [dip.exec.resultHandler] act.compute_NL9Ayu9Z_NP - Error file found, trying to throw it: /opt/DATA_DIR/jobs/SPOPTIMIZATION/Build_projection_output__NP__2022-12-16T02-21-49.030/compute_NL9Ayu9Z_NP/python-recipe/pyout455zZxomC3EN/error.json
[2022/12/16-02:21:52.841] [FRT-37-FlowRunnable] [INFO] [dip.exec.resultHandler] act.compute_NL9Ayu9Z_NP - Raw error is{"errorType":"\u003cclass \u0027TypeError\u0027\u003e","message":"copy() got an unexpected keyword argument \u0027deep\u0027","detailedMessage":"At line 81: \u003cclass \u0027TypeError\u0027\u003e: copy() got an unexpected keyword argument \u0027deep\u0027","stackTrace":[]}
[2022/12/16-02:21:52.841] [FRT-37-FlowRunnable] [INFO] [dip.exec.resultHandler] act.compute_NL9Ayu9Z_NP - After enrichment of error file, error is: {"errorType":"\u003cclass \u0027TypeError\u0027\u003e","message":"Error in Python process: copy() got an unexpected keyword argument \u0027deep\u0027","detailedMessage":"Error in Python process: At line 81: \u003cclass \u0027TypeError\u0027\u003e: copy() got an unexpected keyword argument \u0027deep\u0027","stackTrace":[]}
[2022/12/16-02:21:52.845] [FRT-37-FlowRunnable] [INFO] [dku.flow.activity] act.compute_NL9Ayu9Z_NP - Run thread failed for activity compute_NL9Ayu9Z_NP
com.dataiku.common.server.APIError$SerializedErrorException: Error in Python process: At line 81: <class 'TypeError'>: copy() got an unexpected keyword argument 'deep'
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileIfPossible(JobExecutionResultHandler.java:106)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileOrLogs(JobExecutionResultHandler.java:39)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileOrLogs(JobExecutionResultHandler.java:34)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.handleExecutionResult(JobExecutionResultHandler.java:26)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:73)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:57)
    at com.dataiku.dip.recipes.code.python.PythonRecipeRunner.run(PythonRecipeRunner.java:72)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2022/12/16-02:21:53.022] [ActivityExecutor-33] [INFO] [dku.flow.activity] running compute_NL9Ayu9Z_NP - activity is finished
[2022/12/16-02:21:53.023] [ActivityExecutor-33] [ERROR] [dku.flow.activity] running compute_NL9Ayu9Z_NP - Activity failed
com.dataiku.common.server.APIError$SerializedErrorException: Error in Python process: At line 81: <class 'TypeError'>: copy() got an unexpected keyword argument 'deep'
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileIfPossible(JobExecutionResultHandler.java:106)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileOrLogs(JobExecutionResultHandler.java:39)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.throwFromErrorFileOrLogs(JobExecutionResultHandler.java:34)
    at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.handleExecutionResult(JobExecutionResultHandler.java:26)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:73)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:57)
    at com.dataiku.dip.recipes.code.python.PythonRecipeRunner.run(PythonRecipeRunner.java:72)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2022/12/16-02:21:53.023] [ActivityExecutor-33] [INFO] [dku.flow.activity] running compute_NL9Ayu9Z_NP - Executing default post-activity lifecycle hook
[2022/12/16-02:21:53.026] [ActivityExecutor-33] [INFO] [dku.flow.activity] running compute_NL9Ayu9Z_NP - Done post-activity tasks

---

Operating system used: ubuntu

Alexandru

Hi @Sup
,


The error :
[2022/12/16-02:21:52.610] [null-err-40] [INFO] [dku.utils] - df = df.copy(deep=False)
[2022/12/16-02:21:52.610] [null-err-40] [INFO] [dku.utils] - TypeError: copy() got an unexpected keyword argument 'deep'
This seems related to the panda's version and should be fixed with pandas 1.0.1+
Note you can use pandas 1.1-1.3 in DSS 0.0.4+ https://doc.dataiku.com/dss/latest/release_notes/10.0.html#version-10-0-4-march-7th-2022

Can you try updating the panda's version if that doesn't help please raise a support ticket with the job diagnostics and code env diagnostics.

Thanks