Forcast Plugin run error

Options
bilgicsin
bilgicsin Partner, Registered Posts: 5 Partner
edited July 16 in Using Dataiku

Hi,

I am trying to use Forecast plugin on Dataiku. I installed plugin and created a dataiku managed code environment successfully. When I tried the run plugin I got the error below:

[2021/07/28-14:11:38.745] [FRT-35-FlowRunnable] [INFO] [dku.usage.computeresource.jek] act.compute_d3ayBhqh_NP - Reporting completion of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"admin","projectKey":"time_series_prediction","jobId":"Build_time_series_2021-07-28T11-11-33.947","activityId":"compute_d3ayBhqh_NP","activityType":"recipe","recipeType":"CustomCode_timeseries-forecast-1-train-evaluate","recipeName":"compute_d3ayBhqh"},"type":"LOCAL_PROCESS","id":"SetvSUeBUuBLHdVd","startTime":1627470697569,"endTime":1627470698744,"localProcess":{"pid":4855,"commandName":"/data/DATA_DIR/code-envs/python/plugin_timeseries-forecast_managed/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":121,"vmRSSMB":4,"vmHWMMB":4,"vmRSSAnonMB":2,"vmDataMB":2,"vmSizePeakMB":121,"vmRSSPeakMB":4,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}}
[2021/07/28-14:11:38.748] [FRT-35-FlowRunnable] [INFO] [dku.flow.activity] act.compute_d3ayBhqh_NP - Run thread failed for activity compute_d3ayBhqh_NP
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
    at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
    at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2021/07/28-14:11:38.909] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - activity is finished
[2021/07/28-14:11:38.909] [ActivityExecutor-29] [ERROR] [dku.flow.activity] running compute_d3ayBhqh_NP - Activity failed
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
    at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.handleExecutionResult(AbstractCodeBasedActivityRunner.java:189)
    at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:103)
    at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
    at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
    at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[2021/07/28-14:11:38.910] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Executing default post-activity lifecycle hook
[2021/07/28-14:11:38.914] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_performance
[2021/07/28-14:11:38.915] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Removing samples for time_series_prediction.time_series_evaluation
[2021/07/28-14:11:38.916] [ActivityExecutor-29] [INFO] [dku.flow.activity] running compute_d3ayBhqh_NP - Done post-activity tasks

I am using Dataiku on a demo environment. I have to demonstrate this plugin to customer. So I need help

Thanks in advance

· Share on FacebookShare on Twitter

Best Answer

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Hi,

    As described here as well https://community.dataiku.com/t5/Using-Dataiku-DSS/Job-failed-The-Python-process-failed-exit-code-132/m-p/18252

    The Python process failed (exit code: 132) errors suggests the CPU instruction set required for the plugin is not available on your demo machine.

    To confirm this you can check the output of: cat /proc/cpuinfo

    To use the forecast plugin you may need to install DSS natively on a machine that supports the AVX instructions set.

    · Share on FacebookShare on Twitter
  • bilgicsin
    bilgicsin Partner, Registered Posts: 5 Partner
    edited July 17
    Options

    Hi @AlexT
    ,

    Thanks for the reply!

    I am aware of this discussion but I think AVX is not the issue for me. My CPU model is Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz and it is seems to be supported avx:

    https://ark.intel.com/content/www/tr/tr/ark/products/64589/intel-xeon-processor-e5-2667-15m-cache-2-90-ghz-8-00-gt-s-intel-qpi.html

    Am I missing something?

    I will share my CPU information below:

    [root@dataiku spec]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz
stepping        : 7
microcode       : 0x718
cpu MHz         : 2893.028
cache size      : 15360 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm ssbd ibrs ibpb stibp tsc_adjust arat md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities
bogomips        : 5786.05
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
    · Share on FacebookShare on Twitter
  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Indeed avx is listed. It could be some other instruction set that is missing. Since the CPU( E5-2667) itself is quite old, circa 2012. Can you confirm where you are running this DSS instance?

    What OS version? What platform, Virtualbox? VMWare?

    Are you able to reproduce the issue on any other machine?

    · Share on FacebookShare on Twitter
  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    If I am not mistaken, more recent TensorFlow versions actually require AVX2 now. Your CPU only has AVX. As suggested in an earlier thread if this is just for demo purposes you may be able to simply downgrade the TensorFlow version.

    Looks related to https://github.com/tensorflow/tensorflow/issues/45744 which seems to suggest that AVX is again supported in a later TensorFlow, but for 1.6 or higher you need AVX2.

    · Share on FacebookShare on Twitter
  • bilgicsin
    bilgicsin Partner, Registered Posts: 5 Partner
    Options

    Hi @AlexT

    We have a datacenter in our company for demo purposes. We control VMs via VMWARE.

    The VM that Dataiku installed is Centos 7 64 bit machine.

    We have another VM, which is installed Dataiku, and We get same error with the plugin.

    On the path "~/DATA_DIR/plugins/installed/timeseries-forecast/code-env/python/spec" , there is a requirement.txt like :

    mxnet==1.7.0.post1
    gluonts==0.6.4
    dill==0.3.3
    pmdarima==1.8.0
    threadpoolctl==2.1.0
    statsmodels==0.12.2

    I could not see a library about tensorflow. Am I looking the wrong location? Is there any better way to use earlier version of tensorflow on Forecast plugin?

    · Share on FacebookShare on Twitter
  • bilgicsin
    bilgicsin Partner, Registered Posts: 5 Partner
    Options

    Unfortunately no, We have to run it on our local machines I guess. I tried to downgrade packages but it causes other dependency problems. I think there is nothing to do.

    Thanks for help @AlexT
    !

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…