Name External Conda environment - how to use containerized execution ?
We've created a 'Name External Conda environment' in DSS and we would like to use this code environment in a python recipe with a containerized execution.
How to setup this 'Name External Conda environment' with containerized execution as the option to define the container is not available from DSS screen like for the other code environments ?
Annie
Best Answer
-
Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 DataikerOptionsHi @rona
If we are talking about Python3.8, you can only make this code env build from a regular (non-conda) python binary by passing over the path to the custom python interpreter:
Conda options will be limited to Python2.7, 3.5, 3.6, and 3.7 (for DSS9). Please note you will not be able to create a containerized code env that was built with a custom python interpreter.
Answers
-
Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 DataikerOptions
Hi @rona
It's not possible to use such environments in the containerized executions because DSS is not aware of what packages are actually installed in that code env as the user installs the packages manually outside of DSS.
If you need to have a code env that you can use in containerized execution (and built with conda) you will need to create a managed code env with "Use conda" checkbox enabled:
You will be able to enable containerized execution for it:
Hope you will find this information helpful.
-
Thanks a lot for this feedback.
We tried to use the code environment building with conda and for the package rdkit we had to add manually the channel conda-forge.
conda activate ... dss/code-envs/python/dssconda-py37
conda config --append channels conda-forgePlease, do you know if there is another way to add this configuration directly with the GUI of DSS ?
When we try to use the containerization with this code environment, there is an issue by doing this action outside DSS.
Annie
-
Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 DataikerOptions
Hi @rona
Yes, you can add an additional conda channel by going to managed code env -> General tab where you need to uncheck "Inherit global settings" and add two lines to the Extra options for 'conda install':
This will add a conda-forge channel so "rdkit" package will be available for the installation.
-
Thanks a lot ! it works fine this way both on local and with containerized execution.
Additional question : can we build code environment with conda option in DSS with python 3.8 ? (even it's not supported in case of issues).
Annie
-
Thanks a lot Sergey for this information. It's clear now.
Annie
-
Adding "-c conda-forge" in two separate rows does not work for me. Any idea what could be the cause?
