Is it possible to install a python package directly to a code environment through command-line?

SallmanGhanem
SallmanGhanem Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 5 ✭✭✭✭

I have a custom python 3.6 code environment in dataiku. I know you can extend the list of packages to install through Requested Packages (PIP). However, the package I want to install is outside of pip so I am using the command below as suggested:

-e /directory/to/package 

However, since dataiku runs with the dataiku user I am getting permission failed. It seems like I need to execute the pip install as root user. Therefore, I am wondering is it possible to navigate to the custom code env in the root folder and execute a pip install there as root? If so where would I find my custom code environments.

Ps. I did try this inside Requested Packages (PIP) which should allow you to circumvent the root requirements but that didn't work...

--user -e /directory/to/package

EDIT 1: I see there is a desc.json file under /code-envs/desc/python/myenv in the data directory which has the following:

{
  "pythonInterpreter": "PYTHON36",
  "installCorePackages": true,
  "corePackagesSet": "PANDAS10",
  "installJupyterSupport": true,
  "conda": false,
  "creationTag": {
    "versionNumber": 0,
    "lastModifiedBy": {
      "login": "admin"
    },
    "lastModifiedOn": 1612265307501
  },
  "versionTag": {
    "versionNumber": 0,
    "lastModifiedBy": {
      "login": "admin"
    },
    "lastModifiedOn": 1612265307501
  },
  "deploymentMode": "DESIGN_MANAGED",
  "envSettings": {
    "inheritGlobalSettings": true,
    "condaInstallExtraOptions": [],
    "condaCreateExtraOptions": [],
    "pipInstallExtraOptions": [],
    "virtualenvCreateExtraOptions": [],
    "cranMirrorURL": "https://cloud.r-project.org"
  },

can the "pipInstallExtraOptions" be edited somehow to execute as root?

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Answer ✓

    Hi,

    the `pipInstallExtraOptions` is where you add command flags, such as `--proxy` and stuff, for `pip` commands. You can edit in the DSS UI in Administration > Settings > Misc.

    But you should not attempt to run pip for code envs as the root user, that would most likely make the code env unusable later on, since all commands issued by DSS on the code env will be with the dataiku user. You should instead make the /directory/to/package accessible to the dataiku user, and put it in the required packages. The list of required packages accepts package names, git repositories urls, paths to tar.gz files of packages, and even paths to the folder of the source code of a package.

    If using root is mandatory, then you can make a "non-managed path" code env, but then you'll have to manually setup the python virtualenv in the path of the code env, and make sure it's accessible to the dataiku user

Setup Info
    Tags
      Help me…