Plugin Python Code environment error - Pandas Version conflict

NN

Hi,

I am trying to create a plugin managed python environment for a custom recipe.

The problem i face is that while setting up the environment i get the error

Updating code environment according to spec 
...Installing from Pip requirementsERROR: 
Cannot install pandas<1.1 and >=1.0 and pandas==0.23.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependenciesThe conflict is caused by:  
The user requested pandas<1.1 and >=1.0    
The user requested pandas==0.23.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

But when we create a DESIGN_MANAGED python code environment with similar settings there doesnt seem to be a conflict there.

Can someone help if they have faced a similar issue ?

Some additional details have been shared below.

Dataiku version - 8.0.1

dataiku environment - Python36 / Managed by DSS / No containers

desc.json ->

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"installJupyterSupport": false
}

Requirements.txt ->

pandas>=1.0,<1.1

Ignacio_Toledo

Hi @NN
,

I just made a test and this worked for me: just add a new entry to the desc.json like this:

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"corePackagesSet": "PANDAS10",
"installJupyterSupport": false
}

After regenerating the plugin code environment there were no problems with using pandas 1.

Hope this helps!

NN

That Worked perfectly Thanks a lot.

clayms

Still a conflict

ERROR: Cannot install pandas<1.1 and >=1.0 and pandas<1.4 and >=1.3 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

The following fixes this, but we should not have to do this.

Why can't it just install the version that is specified in the requirements.txt file?

Where is this documented?

"corePackagesSet": "PANDAS13",

Ignacio_Toledo

Hi @clayms
,

I guess it has to do with the dataiku api dependencies with pandas, which also explains why pandas versions 1.2 and 1.3 are only available for a python 3.7 and above version.

Hope this helps

Ignacio 

clayms

Thanks @Ignacio_Toledo


That makes sense, and I am aware of the restricted options when building a code environment.

That still leaves the question: Where is this documented?

Ignacio_Toledo

It is a good question! I'm trying to find out how did I come to that solution, and I'm guessing it was some reverse-engineering with some exported environment from dataiku, where I found out about the corePackagesSet values. Or maybe from a plugin that I converted into a development plugin?

Anyhow, there is documentation in the knowledge base and the dataiku docs, but not with this level of detail I think...