Job failed: The Python process failed (exit code: 132)

aliciacorain
aliciacorain Registered Posts: 2 ✭✭✭

I got an error when running the train & evaluate forecasting models. Attached are the logs.

[15:24:12] [INFO] [dku.flow.activity] - Run thread failed for activity compute_qKug334Y_NP
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.handleExecutionResult(JobExecutionResultHandler.java:26)
at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:71)
at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)
[15:24:12] [INFO] [dku.flow.activity] running compute_qKug334Y_NP - activity is finished
[15:24:12] [ERROR] [dku.flow.activity] running compute_qKug334Y_NP - Activity failed
com.dataiku.dip.exceptions.ProcessDiedException: The Python process failed (exit code: 132). More info might be available in the logs.
at com.dataiku.dip.dataflow.common.CodeBasedThingHelper.throwSubprocessError(CodeBasedThingHelper.java:23)
at com.dataiku.dip.dataflow.exec.JobExecutionResultHandler.handleExecutionResult(JobExecutionResultHandler.java:26)
at com.dataiku.dip.dataflow.exec.AbstractCodeBasedActivityRunner.execute(AbstractCodeBasedActivityRunner.java:71)
at com.dataiku.dip.dataflow.exec.AbstractPythonRecipeRunner.executeScript(AbstractPythonRecipeRunner.java:48)
at com.dataiku.dip.recipes.customcode.CustomPythonRecipeRunner.run(CustomPythonRecipeRunner.java:71)
at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:374)

Any ideas on how to solve it?

Thanks in advance!

Answers

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi @aliciacorain

    Exit code 132 can have two potential root causes:
    - Bug in the program itself
    - CPU cannot run this program, because this program was compiled for a higher processor.
    Note: “the program” here is a python library, not DSS.
    Can you provide more details on this error? Is this a python code recipe? Are you trying to train the model?
  • aliciacorain
    aliciacorain Registered Posts: 2 ✭✭✭

    Hi @sergeyd
    ,

    I was trying to train the model and it's the Train and evaluate forecasting models recipe from Dataiku Forecast plugin. Does this help?

    Thanks!

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    edited July 17

    What is the result of this command executed in DSS CLI:

    cat /proc/cpuinfo

    ?

  • bilgicsin
    bilgicsin Partner, Registered Posts: 5 Partner

    Hi,

    I am getting the same error, did you solve the problem? @aliciacorain

  • Sergey9909
    Sergey9909 Registered Posts: 1 ✭✭✭

    Me too =\

    I Use it by virtualBox and got it then try to use plagin for time series forecast

  • xavme
    xavme Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 8 ✭✭✭✭

    Hi, I also have the same issue;

    to answer @sergeyd
    's question, here is the result from cat /proc/cpuinfo that were requested by

    processor : 0
    vendor_id : GenuineIntel
    cpu family : 6
    model : 85
    model name : Intel Xeon Processor (Cascadelake)
    stepping : 5
    microcode : 0x1
    cpu MHz : 2593.894
    cache size : 4096 KB
    physical id : 0
    siblings : 2
    core id : 0
    cpu cores : 2
    apicid : 0
    initial apicid : 0
    fpu : yes
    fpu_exception : yes
    cpuid level : 13
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 hypervisor lahf_lm pti pku ospke
    bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
    bogomips : 5187.78
    clflush size : 64
    cache_alignment : 64
    address sizes : 46 bits physical, 48 bits virtual
    power management:

    processor : 1
    vendor_id : GenuineIntel
    cpu family : 6
    model : 85
    model name : Intel Xeon Processor (Cascadelake)
    stepping : 5
    microcode : 0x1
    cpu MHz : 2593.894
    cache size : 4096 KB
    physical id : 0
    siblings : 2
    core id : 1
    cpu cores : 2
    apicid : 1
    initial apicid : 1
    fpu : yes
    fpu_exception : yes
    cpuid level : 13
    wp : yes
    flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 hypervisor lahf_lm pti pku ospke
    bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
    bogomips : 5187.78
    clflush size : 64
    cache_alignment : 64
    address sizes : 46 bits physical, 48 bits virtual
    power management:

    Thanks!

    Xavier

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi @xavme

    Thanks for the provided output. I do suspect the CPU type is the root cause here as it doesn't have AVX instructions set.
    Do you have any other machine with a more recent CPU to test this job?
  • xavme
    xavme Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 8 ✭✭✭✭

    Yes @sergeyd
    , I tried exporting the project on my local install (virtualbox on windows) and it works; it might be the CPU, I thought it would be due to server / ubuntu / dss configuration.

    I'm actually seeing the error on the dataiku server that we installed to share the project with the team. is there a way to configure the server to be able to use the forecast plugin, even with the current CPU?

    thanks

    Xavier

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    edited July 17

    Hi @xavme

    Technically, there is a possibility with lowering the versions of the Python packages required for this plugin:

    mxnet==1.7.0.post1
    gluonts==0.7.6
    dill==0.3.3
    pmdarima==1.8.0
    threadpoolctl==2.1.0
    statsmodels==0.12.2

    but there is a high chance that plugin recipes will not work due to missing classes or functions that are present in the package versions we install.

  • xavme
    xavme Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 8 ✭✭✭✭

    ok, seems like i will have to do time series only on my local VM...

    thanks for helping!

    xavier

  • Ahmed_TLILI
    Ahmed_TLILI Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 2 Partner

    Hello @sergeyd
    ,

    I am running through the same error and I did the checking the CPU cmd but I have got a recent CPU (intel xenon gold 6140 (2017) and it got AVX enabled).

    What do you think the problem is?

    P.S: Same log details and I am trying to run the forecasting recipe (train and evaluate models)

Setup Info
    Tags
      Help me…