Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Plugin Python Code environment error - Pandas Version conflict
Added on December 15, 2020 8:04PM
Likes: 0
Replies: 6
NN
Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 145 Neuron
Hi,
I am trying to create a plugin managed python environment for a custom recipe.
The problem i face is that while setting up the environment i get the error
Updating code environment according to spec
...Installing from Pip requirementsERROR:
Cannot install pandas<1.1 and >=1.0 and pandas==0.23.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependenciesThe conflict is caused by:
The user requested pandas<1.1 and >=1.0
The user requested pandas==0.23.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
But when we create a DESIGN_MANAGED python code environment with similar settings there doesnt seem to be a conflict there.
Can someone help if they have faced a similar issue ?
Some additional details have been shared below.
Dataiku version - 8.0.1
dataiku environment - Python36 / Managed by DSS / No containers
desc.json ->
{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"installJupyterSupport": false
}
Requirements.txt ->
pandas>=1.0,<1.1
Best Answer
-
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 419 NeuronHi @NN
,I just made a test and this worked for me: just add a new entry to the desc.json like this:
{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"corePackagesSet": "PANDAS10",
"installJupyterSupport": false
}After regenerating the plugin code environment there were no problems with using pandas 1.
Hope this helps!
Answers
-
That Worked perfectly Thanks a lot.
-
Still a conflict
ERROR: Cannot install pandas<1.1 and >=1.0 and pandas<1.4 and >=1.3 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
The following fixes this, but we should not have to do this.Why can't it just install the version that is specified in the requirements.txt file?
Where is this documented?"corePackagesSet": "PANDAS13",
-
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 419 Neuron
Hi @clayms
,I guess it has to do with the dataiku api dependencies with pandas, which also explains why pandas versions 1.2 and 1.3 are only available for a python 3.7 and above version.
Hope this helps
Ignacio
-
Thanks @Ignacio_Toledo
That makes sense, and I am aware of the restricted options when building a code environment.That still leaves the question: Where is this documented?
-
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 419 Neuron
It is a good question! I'm trying to find out how did I come to that solution, and I'm guessing it was some reverse-engineering with some exported environment from dataiku, where I found out about the corePackagesSet values. Or maybe from a plugin that I converted into a development plugin?
Anyhow, there is documentation in the knowledge base and the dataiku docs, but not with this level of detail I think...
