Plugin Python Code environment error - Pandas Version conflict

Options
NN
NN Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 145 Neuron

Hi,

I am trying to create a plugin managed python environment for a custom recipe.

The problem i face is that while setting up the environment i get the error

Updating code environment according to spec 
...
Installing from Pip requirementsERROR:
Cannot install pandas<1.1 and >=1.0 and pandas==0.23.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependenciesThe conflict is caused by:
The user requested pandas<1.1 and >=1.0

The user requested pandas==0.23.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

But when we create a DESIGN_MANAGED python code environment with similar settings there doesnt seem to be a conflict there.

Can someone help if they have faced a similar issue ?

Some additional details have been shared below.

Dataiku version - 8.0.1

dataiku environment - Python36 / Managed by DSS / No containers

desc.json ->

{
"acceptedPythonInterpreters": ["PYTHON36"],
"forceConda": false,
"installCorePackages": true,
"installJupyterSupport": false
}

Requirements.txt ->

pandas>=1.0,<1.1

Best Answer

Answers

  • NN
    NN Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 145 Neuron
    Options

    That Worked perfectly Thanks a lot.

  • clayms
    clayms Registered Posts: 52 ✭✭✭✭
    edited July 17
    Options

    Still a conflict

    ERROR: Cannot install pandas<1.1 and >=1.0 and pandas<1.4 and >=1.3 because these package versions have conflicting dependencies.
    ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies


    The following fixes this, but we should not have to do this.

    Why can't it just install the version that is specified in the requirements.txt file?


    Where is this documented?

    "corePackagesSet": "PANDAS13",

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Hi @clayms
    ,

    I guess it has to do with the dataiku api dependencies with pandas, which also explains why pandas versions 1.2 and 1.3 are only available for a python 3.7 and above version.

    dataiku python and pandas.png

    Hope this helps

    Ignacio 

  • clayms
    clayms Registered Posts: 52 ✭✭✭✭
    Options

    Thanks @Ignacio_Toledo


    That makes sense, and I am aware of the restricted options when building a code environment.

    That still leaves the question: Where is this documented?

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    It is a good question! I'm trying to find out how did I come to that solution, and I'm guessing it was some reverse-engineering with some exported environment from dataiku, where I found out about the corePackagesSet values. Or maybe from a plugin that I converted into a development plugin?

    Anyhow, there is documentation in the knowledge base and the dataiku docs, but not with this level of detail I think...

Setup Info
    Tags
      Help me…