Plugin Python Code environment error - Pandas Version conflict

Solved!
NN
Plugin Python Code environment error - Pandas Version conflict

Hi,

I am trying to create a plugin managed python environment for a custom recipe.

The problem i face is that while setting up the environment i get the error 

Updating code environment according to spec 
...
Installing from Pip requirementsERROR:
Cannot install pandas<1.1 and >=1.0 and pandas==0.23.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependenciesThe conflict is caused by:
The user requested pandas<1.1 and >=1.0

The user requested pandas==0.23.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

But when we create a DESIGN_MANAGED python code environment with similar settings there doesnt seem to be a conflict there.

Can someone help if they have faced a similar issue ?

Some additional details have been shared below.

Dataiku version - 8.0.1

dataiku environment - Python36 / Managed by DSS / No containers

desc.json ->

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"installJupyterSupport": false
}

Requirements.txt ->

pandas>=1.0,<1.1

 

0 Kudos
1 Solution
Ignacio_Toledo

Hi @NN,

I just made a test and this worked for me: just add a new entry to the desc.json like this:

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"corePackagesSet": "PANDAS10",
"installJupyterSupport": false
}

After regenerating the plugin code environment there were no problems with using pandas 1.

Hope this helps! 

View solution in original post

6 Replies
Ignacio_Toledo

Hi @NN,

I just made a test and this worked for me: just add a new entry to the desc.json like this:

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"corePackagesSet": "PANDAS10",
"installJupyterSupport": false
}

After regenerating the plugin code environment there were no problems with using pandas 1.

Hope this helps! 

NN
Author

That Worked perfectly Thanks a lot.

clayms
Level 3

Still a conflict 

ERROR: Cannot install pandas<1.1 and >=1.0 and pandas<1.4 and >=1.3 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies


The following fixes this, but we should not have to do this. 

Why can't it just install the version that is specified in the requirements.txt file?


Where is this documented? 

"corePackagesSet": "PANDAS13",

 

0 Kudos
Ignacio_Toledo

Hi @clayms,

I guess it has to do with the dataiku api dependencies with pandas, which also explains why pandas versions 1.2 and 1.3 are only available for a python 3.7 and above version.

dataiku python and pandas.png

Hope this helps

Ignacioโ€ƒ

0 Kudos
clayms
Level 3

Thanks @Ignacio_Toledo 

That makes sense, and I am aware of the restricted options when building a code environment. 

That still leaves the question: Where is this documented?

0 Kudos
Ignacio_Toledo

It is a good question! I'm trying to find out how did I come to that solution, and I'm guessing it was some reverse-engineering with some exported environment from dataiku, where I found out about the corePackagesSet values. Or maybe from a plugin that I converted into a development plugin?

Anyhow, there is documentation in the knowledge base and the dataiku docs, but not with this level of detail I think...