Forcast Plugin run error
Hi,
I am trying to use Forecast plugin on Dataiku. I installed plugin and created a dataiku managed code environment successfully. When I tried the run plugin I got the error below:
[2021/07/28-14:11:38.745] [FRT-35-FlowRunnable] [INFO] [dku.usage.computeresource.jek] act.compute_d3ayBhqh_NP - Reporting completion of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"admin","projectKey":"time_series_prediction","jobId":"Build_time_series_2021-07-28T11-11-33.947","activityId":"compute_d3ayBhqh_NP","activityType":"recipe","recipeType":"CustomCode_timeseries-forecast-1-train-evaluate","recipeName":"compute_d3ayBhqh"},"type":"LOCAL_PROCESS","id":"SetvSUeBUuBLHdVd","startTime":1627470697569,"endTime":1627470698744,"localProcess":{"pid":4855,"commandName":"/data/DATA_DIR/code-envs/python/plugin_timeseries-forecast_managed/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":121,"vmRSSMB":4,"vmHWMMB":4,"vmRSSAnonMB":2,"vmDataMB":2,"vmSizePeakMB":121,"vmRSSPeakMB":4,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}} [2021/07/28-14:11:38.748] [FRT-35-FlowRunnable] [INFO] [dku.flow.activity] act.compute_d3ayBhqh_NP - Run thread failed for activity compute_d3ayBhqh_NP com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs. at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103) at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48) at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374) [2021/07/28-14:11:38.909] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - activity is finished [2021/07/28-14:11:38.909] [ActivityExecutor-29] [ERROR] [dku.flow.activity] running compute_d3ayBhqh_NP - Activity failed com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs. at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189) at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103) at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48) at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374) [2021/07/28-14:11:38.910] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Executing default post-activity lifecycle hook [2021/07/28-14:11:38.914] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_performance [2021/07/28-14:11:38.915] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_evaluation [2021/07/28-14:11:38.916] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Done post-activity tasks
I am using Dataiku on a demo environment. I have to demonstrate this plugin to customer. So I need help
Thanks in advance
Best Answer
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker
Looks like other packages mxnet package actually requires AVX2 as well. Do you have any machines with AVX2 instruction set available?
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker
Hi,
As described here as well https://community.dataiku.com/t5/Using-Dataiku-DSS/Job-failed-The-Python-process-failed-exit-code-132/m-p/18252
The Python process failed (exit code: 132) errors suggests the CPU instruction set required for the plugin is not available on your demo machine.
To confirm this you can check the output of: cat /proc/cpuinfo
To use the forecast plugin you may need to install DSS natively on a machine that supports the AVX instructions set.
-
Hi @AlexT
,Thanks for the reply!
I am aware of this discussion but I think AVX is not the issue for me. My CPU model is Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz and it is seems to be supported avx:
Am I missing something?
I will share my CPU information below:
[root@dataiku spec]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 45 model name : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz stepping : 7 microcode : 0x718 cpu MHz : 2893.028 cache size : 15360 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm ssbd ibrs ibpb stibp tsc_adjust arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities bogomips : 5786.05 clflush size : 64 cache_alignment : 64 address sizes : 43 bits physical, 48 bits virtual
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker
Indeed avx is listed. It could be some other instruction set that is missing. Since the CPU( E5-2667) itself is quite old, circa 2012. Can you confirm where you are running this DSS instance?
What OS version? What platform, Virtualbox? VMWare?
Are you able to reproduce the issue on any other machine?
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker
If I am not mistaken, more recent TensorFlow versions actually require AVX2 now. Your CPU only has AVX. As suggested in an earlier thread if this is just for demo purposes you may be able to simply downgrade the TensorFlow version.
Looks related to https://github.com/tensorflow/tensorflow/issues/45744 which seems to suggest that AVX is again supported in a later TensorFlow, but for 1.6 or higher you need AVX2.
-
Hi @AlexT
We have a datacenter in our company for demo purposes. We control VMs via VMWARE.
The VM that Dataiku installed is Centos 7 64 bit machine.
We have another VM, which is installed Dataiku, and We get same error with the plugin.
On the path "~/DATA_DIR/plugins/installed/timeseries-forecast/code-env/python/spec" , there is a requirement.txt like :
mxnet==1.7.0.post1
gluonts==0.6.4
dill==0.3.3
pmdarima==1.8.0
threadpoolctl==2.1.0
statsmodels==0.12.2I could not see a library about tensorflow. Am I looking the wrong location? Is there any better way to use earlier version of tensorflow on Forecast plugin?
-
Unfortunately no, We have to run it on our local machines I guess. I tried to downgrade packages but it causes other dependency problems. I think there is nothing to do.
Thanks for help @AlexT
!