Python recipe failing in a Python env

[danusio]

Hey guys!

I set a Python env in order to install specific packages, but the following recipe is failing when executed in such an env:

import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
df_train_asis = dataiku.Dataset("df_train_asis")
df_train_asis_df = df_train_asis.get_dataframe()
df_test_asis = dataiku.Dataset("df_test_asis")
df_test_asis_df = df_test_asis.get_dataframe()


# Compute recipe outputs
# TODO: Write here your actual code that computes the outputs
# NB: DSS supports several kinds of APIs for reading and writing data. Please see doc.

df_results_asis_df = df_train_asis_df # Compute a Pandas dataframe to write into df_results_asis


# Write recipe outputs
df_results_asis = dataiku.Dataset("df_results_asis")
df_results_asis.write_with_schema(df_results_asis_df)

The error message:

Job failed: The Python process failed (exit code: 1)

Error type:com.dataiku.dip.exceptions.ProcessDiedException

The script works fine when run in the builtin DSS env. Any idea?

---

Operating system used: MacOS

---

Operating system used: MacOS

[Turribeach]

The recipe looks fine, how exactly did you setup the Python environment? What Python version is using?

[danusio]

I'm using Python 3.9. The only different thing I set was to insert joblib, category_encoders and imblearn in Requested packages (Pip), and everything went well in installation. Current installed packages:

backcall==0.2.0
category-encoders==2.6.2
certifi==2023.7.22
charset-normalizer==3.3.0
decorator==5.1.1
idna==3.4
imbalanced-learn==0.11.0
imblearn==0.0
ipykernel==4.8.2
ipython==7.34.0
ipython-genutils==0.2.0
jedi==0.19.1
joblib==1.3.2
jupyter-client==5.2.4
jupyter_core==4.11.2
matplotlib-inline==0.1.6
numpy==1.23.5
packaging==23.2
pandas==1.1.5
parso==0.8.3
patsy==0.5.3
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.39
ptyprocess==0.7.0
Pygments==2.16.1
python-dateutil==2.8.1
pytz==2020.5
pyzmq==22.3.0
requests==2.31.0
scikit-learn==1.3.1
scipy==1.11.3
simplegeneric==0.8.1
six==1.16.0
statsmodels==0.14.0
threadpoolctl==3.2.0
tornado==5.1.1
traitlets==4.3.3
urllib3==2.0.6
wcwidth==0.2.8

[Turribeach]

Make sure python 3.9 is allowed to open in MacOS Settings 

[Turribeach]

How did you install Python 3.9 in your Mac?

[danusio · Accepted solution]

I tried in a online Dataiku license and got the same, but the problem was the Python version indeed: only 3.6 works properly.

[Turribeach]

I ran this recipe on my Mac on 3.9 and works fine. Your code doesn't look bad to me. Can you remove one of the inputs to test?

# -*- coding: utf-8 -*-
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
customers = dataiku.Dataset("customers")
customers_df = customers.get_dataframe()


# Compute recipe outputs from inputs
# TODO: Replace this part by your actual code that computes the output, as a Pandas dataframe
# NB: DSS also supports other kinds of APIs for reading and writing data. Please see doc.

customers_out_df = customers_df # For this sample code, simply copy input to output


# Write recipe outputs
customers_out = dataiku.Dataset("customers_out")
customers_out.write_with_schema(customers_out_df)

[danusio]

Done, I got the same. Code:

# -*- coding: utf-8 -*-
import dataiku
import pandas as pd, numpy as np
from dataiku import pandasutils as pdu

# Read recipe inputs
df_train_asis = dataiku.Dataset("df_train_asis")
df_train_asis_df = df_train_asis.get_dataframe()


# Compute recipe outputs
# TODO: Write here your actual code that computes the outputs
# NB: DSS supports several kinds of APIs for reading and writing data. Please see doc.

df_check_env_df = df_train_asis_df # Compute a Pandas dataframe to write into df_check_env


# Write recipe outputs
df_check_env = dataiku.Dataset("df_check_env")
df_check_env.write_with_schema(df_check_env_df)

[Turribeach]

What connection are your datasets on?

[danusio]

@Turribeach Amazon S3.

[Turribeach]

What about if you move your datasets to a file system connection?

[danusio]

@Turribeach I'm not able to do that, I'm using a third-part Dataiku instance. Is it possible that the instance is not set to work properly with Python version > 3.6?

[Turribeach]

Anything is possible but without being an Admin you can't really solve anything. I suggest you take up the issue with your Dataiku Administrator.