Forcast Plugin run error

bilgicsin

Hi,

I am trying to use Forecast plugin on Dataiku. I installed plugin and created a dataiku managed code environment successfully. When I tried the run plugin I got the error below:

[2021/07/28-14:11:38.745] [FRT-35-FlowRunnable] [INFO] [dku.usage.computeresource.jek] act.compute_d3ayBhqh_NP - Reporting completion of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"admin","projectKey":"time_series_prediction","jobId":"Build_time_series_2021-07-28T11-11-33.947","activityId":"compute_d3ayBhqh_NP","activityType":"recipe","recipeType":"CustomCode_timeseries-forecast-1-train-evaluate","recipeName":"compute_d3ayBhqh"},"type":"LOCAL_PROCESS","id":"SetvSUeBUuBLHdVd","startTime":1627470697569,"endTime":1627470698744,"localProcess":{"pid":4855,"commandName":"/data/DATA_DIR/code-envs/python/plugin_timeseries-forecast_managed/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":121,"vmRSSMB":4,"vmHWMMB":4,"vmRSSAnonMB":2,"vmDataMB":2,"vmSizePeakMB":121,"vmRSSPeakMB":4,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}}
[2021/07/28-14:11:38.748] [FRT-35-FlowRunnable] [INFO] [dku.flow.activity] act.compute_d3ayBhqh_NP - Run thread failed for activity compute_d3ayBhqh_NP
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
    at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
    at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2021/07/28-14:11:38.909] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - activity is finished
[2021/07/28-14:11:38.909] [ActivityExecutor-29] [ERROR] [dku.flow.activity] running compute_d3ayBhqh_NP - Activity failed
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
    at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
    at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2021/07/28-14:11:38.910] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Executing default post-activity lifecycle hook
[2021/07/28-14:11:38.914] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_performance
[2021/07/28-14:11:38.915] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_evaluation
[2021/07/28-14:11:38.916] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Done post-activity tasks

I am using Dataiku on a demo environment. I have to demonstrate this plugin to customer. So I need help

Thanks in advance

Alexandru

Looks like other packages mxnet package actually requires AVX2 as well. Do you have any machines with AVX2 instruction set available?

https://github.com/apache/incubator-mxnet/issues/10705

Alexandru

Hi,

As described here as well https://community.dataiku.com/t5/Using-Dataiku-DSS/Job-failed-The-Python-process-failed-exit-code-132/m-p/18252

The Python process failed (exit code: 132) errors suggests the CPU instruction set required for the plugin is not available on your demo machine.

To confirm this you can check the output of: cat /proc/cpuinfo

To use the forecast plugin you may need to install DSS natively on a machine that supports the AVX instructions set.

bilgicsin

Hi @AlexT
,

Thanks for the reply!

I am aware of this discussion but I think AVX is not the issue for me. My CPU model is Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz and it is seems to be supported avx:

https://ark.intel.com/content/www/tr/tr/ark/products/64589/intel-xeon-processor-e5-2667-15m-cache-2-90-ghz-8-00-gt-s-intel-qpi.html

Am I missing something?

I will share my CPU information below:

[root@dataiku spec]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz
stepping        : 7
microcode       : 0x718
cpu MHz         : 2893.028
cache size      : 15360 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm ssbd ibrs ibpb stibp tsc_adjust arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities
bogomips        : 5786.05
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual

Alexandru

Indeed avx is listed. It could be some other instruction set that is missing. Since the CPU( E5-2667) itself is quite old, circa 2012. Can you confirm where you are running this DSS instance?

What OS version? What platform, Virtualbox? VMWare?

Are you able to reproduce the issue on any other machine?

Alexandru

If I am not mistaken, more recent TensorFlow versions actually require AVX2 now. Your CPU only has AVX. As suggested in an earlier thread if this is just for demo purposes you may be able to simply downgrade the TensorFlow version.

Looks related to https://github.com/tensorflow/tensorflow/issues/45744 which seems to suggest that AVX is again supported in a later TensorFlow, but for 1.6 or higher you need AVX2.

bilgicsin

Hi @AlexT

We have a datacenter in our company for demo purposes. We control VMs via VMWARE.

The VM that Dataiku installed is Centos 7 64 bit machine.

We have another VM, which is installed Dataiku, and We get same error with the plugin.

On the path "~/DATA_DIR/plugins/installed/timeseries-forecast/code-env/python/spec" , there is a requirement.txt like :

mxnet==1.7.0.post1
gluonts==0.6.4
dill==0.3.3
pmdarima==1.8.0
threadpoolctl==2.1.0
statsmodels==0.12.2

I could not see a library about tensorflow. Am I looking the wrong location? Is there any better way to use earlier version of tensorflow on Forecast plugin?

bilgicsin

Unfortunately no, We have to run it on our local machines I guess. I tried to downgrade packages but it causes other dependency problems. I think there is nothing to do.

Thanks for help @AlexT
!