Name External Conda environment - how to use containerized execution ?

rona
rona Registered Posts: 52 ✭✭✭✭✭

We've created a 'Name External Conda environment' in DSS and we would like to use this code environment in a python recipe with a containerized execution.

How to setup this 'Name External Conda environment' with containerized execution as the option to define the container is not available from DSS screen like for the other code environments ?

Annie

Best Answer

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
    Answer ✓

    Hi @rona

    If we are talking about Python3.8, you can only make this code env build from a regular (non-conda) python binary by passing over the path to the custom python interpreter:

    Screenshot 2021-07-03 at 00.16.06.png

    Conda options will be limited to Python2.7, 3.5, 3.6, and 3.7 (for DSS9). Please note you will not be able to create a containerized code env that was built with a custom python interpreter.

Answers

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi @rona

    It's not possible to use such environments in the containerized executions because DSS is not aware of what packages are actually installed in that code env as the user installs the packages manually outside of DSS.

    If you need to have a code env that you can use in containerized execution (and built with conda) you will need to create a managed code env with "Use conda" checkbox enabled:

    Screenshot 2021-07-01 at 21.58.37.png

    You will be able to enable containerized execution for it:

    Screenshot 2021-07-01 at 22.00.07.png

    Hope you will find this information helpful.

  • rona
    rona Registered Posts: 52 ✭✭✭✭✭

    Thanks a lot for this feedback.

    We tried to use the code environment building with conda and for the package rdkit we had to add manually the channel conda-forge.

    conda activate ... dss/code-envs/python/dssconda-py37
    conda config --append channels conda-forge

    Please, do you know if there is another way to add this configuration directly with the GUI of DSS ?

    When we try to use the containerization with this code environment, there is an issue by doing this action outside DSS.

    Annie

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi @rona

    Yes, you can add an additional conda channel by going to managed code env -> General tab where you need to uncheck "Inherit global settings" and add two lines to the Extra options for 'conda install':

    Screenshot 2021-07-02 at 14.00.20.png

    This will add a conda-forge channel so "rdkit" package will be available for the installation.

  • rona
    rona Registered Posts: 52 ✭✭✭✭✭

    Thanks a lot ! it works fine this way both on local and with containerized execution.

    Additional question : can we build code environment with conda option in DSS with python 3.8 ? (even it's not supported in case of issues).

    Annie

  • rona
    rona Registered Posts: 52 ✭✭✭✭✭

    Thanks a lot Sergey for this information. It's clear now.

    Annie

  • pbena64
    pbena64 Registered Posts: 11

    Adding "-c conda-forge" in two separate rows does not work for me. Any idea what could be the cause?

Setup Info
    Tags
      Help me…