Update pip to latest 20.x

jax79sg
jax79sg Registered Posts: 25 ✭✭✭✭
in Setup & Configuration

Hi,

How do i ensure that Dataiku uses the latest pip?

Thank you.

· Share on FacebookShare on Twitter

Best Answer

Answers

  • Andrey
    Andrey Dataiker Alumni Posts: 119 ✭✭✭✭✭✭✭

    Hi @jax79sg
    ,

    Whenever you create a new python code environment in DSS it's placed in

    DATA_DIR/code-envs/python/ENV_NAME

    so to upgrade pip you'd need to run

    DATA_DIR/code-envs/python/ENV_NAME/bin/pip install --upgrade pip

    Regards

    · Share on FacebookShare on Twitter
  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @Andrey

    Are there any plans from Dataiku to deal with this issue in a more universal way?

    As background, I think that I have something like 14 DSS Python code environments. With at least 2 Design_MANAGED the rest PLUGIN_MANAGED. And then another small handful of R code environments.

    What are the positive and potentially negative ramifications of going to each of these directories and running?

    pip install --upgrade pip

    Do I have the chance to break a plug-in doing this?

    --Tom

    · Share on FacebookShare on Twitter
  • Andrey
    Andrey Dataiker Alumni Posts: 119 ✭✭✭✭✭✭✭

    Hi Tom,

    It's unlikely that upgrading pip will cause issues with the plugins.

    However, changing versions of the libraries in those environments is more risky. For this reason plugins come with it's own requirements.txt that contains a list of library versions that will work with a given version of the plugin.

    Is there a particular reason why you'd want upgrade pip in all of your environments?

    · Share on FacebookShare on Twitter
  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @Andrey
    ,

    Good to know. With my IT operations hat on we like to keep utilities up to date to generally avoid bugs an vulnerabilities.

    I’m taking from your comment that may not be a good idea in this case. That’s why I’m extending this conversation to get a bit of clarity.

    Related, I think I’ve seen errors recently when working with older existing code environments in dss where pip has thrown out an error and specifically called out the need to update pip. I don’t remember if this error caused the build of the environment not to complete successfully.

    So if I am remembering correctly then I guess this would be about making sure that rebuilds run smoothly. I’m not in a place to test at the moment.

    Has anyone else seen things around the version of pip related to dss?

    · Share on FacebookShare on Twitter
  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    Following up on and bumping this old topic - we've run into pip v 20.* in DSS 9.0.5 taking a ton of time and causing issues because of the newer-ish resolver. I know this has been improved in later versions of pip. Is it good practice then to go into each env and upgrade or is there a more universal way to handle this?

    · Share on FacebookShare on Twitter
  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    rock and roll, thank you as always @sergeyd

    · Share on FacebookShare on Twitter
  • Mahdi_N
    Mahdi_N Dataiku DSS Core Designer, Registered, Frontrunner 2022 Participant Posts: 2 ✭✭✭

    Hi Andrey,

    When running the DATA_DIR/code-envs/python/ENV_NAME/bin/pip install --upgrade pip commend I'm getting an error : -bash : pip : command not found

    I searched the error and one reco was to try pip3 and when trying DATA_DIR/code-envs/python/ENV_NAME/bin/pip3 install --upgrade pip3 - I'm getting a different error : Could not find a version that satisfies the requirement upgrade (from version : none)

    I'm not familiar with Linux - hope you can point me in the right direction !

    · Share on FacebookShare on Twitter
  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    edited July 17

    Hi @Mahdi_N

    Before running you will need to check what binaries are present in the code-env directory. Should work with both (pip and pip3) binaries for py3.x code env: 

    (base) [centos@localhost ~]$ dss/code-envs/python/py36_test/bin/pip install --upgrade pip
Requirement already satisfied: pip in ./dss/code-envs/python/py36_test/lib/python3.6/site-packages (21.3.1)
(base) [centos@localhost ~]$ dss/code-envs/python/py36_test/bin/pip3 install --upgrade pip
Requirement already satisfied: pip in ./dss/code-envs/python/py36_test/lib/python3.6/site-packages (21.3.1)

    Also, please check the solution to this post that will make automatically the latest possible pip version to the corresponding python virtualenv version.

    · Share on FacebookShare on Twitter
  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    Confirmed this solution worked well for us on 9.0.5

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…