Is it possible to install a python package directly to a code environment through command-line?

Solved!
SallmanGhanem
Level 2
Is it possible to install a python package directly to a code environment through command-line?

I have a custom python 3.6 code environment in dataiku. I know you can extend the list of packages to install through Requested Packages (PIP). However, the package I want to install is outside of pip so  I am using the command below as suggested:

 

-e /directory/to/package 

 

 However, since dataiku runs with the dataiku user I am getting permission failed. It seems like I need to execute the pip install as root user. Therefore, I am wondering is it possible to navigate to the custom code env in the root folder and execute a pip install there as root? If so where would I find my custom code environments.

 

Ps. I did try this inside Requested Packages (PIP) which should allow you to circumvent the root requirements but that didn't work...

 

--user -e /directory/to/package

 

 EDIT 1: I see there is a desc.json file under /code-envs/desc/python/myenv in the data directory which has the following:

{
  "pythonInterpreter": "PYTHON36",
  "installCorePackages": true,
  "corePackagesSet": "PANDAS10",
  "installJupyterSupport": true,
  "conda": false,
  "creationTag": {
    "versionNumber": 0,
    "lastModifiedBy": {
      "login": "admin"
    },
    "lastModifiedOn": 1612265307501
  },
  "versionTag": {
    "versionNumber": 0,
    "lastModifiedBy": {
      "login": "admin"
    },
    "lastModifiedOn": 1612265307501
  },
  "deploymentMode": "DESIGN_MANAGED",
  "envSettings": {
    "inheritGlobalSettings": true,
    "condaInstallExtraOptions": [],
    "condaCreateExtraOptions": [],
    "pipInstallExtraOptions": [],
    "virtualenvCreateExtraOptions": [],
    "cranMirrorURL": "https://cloud.r-project.org"
  },

can the "pipInstallExtraOptions" be edited somehow to execute as root?

0 Kudos
1 Solution
fchataigner2
Dataiker

Hi, 

the `pipInstallExtraOptions` is where you add command flags, such as `--proxy` and stuff, for `pip` commands. You can edit in the DSS UI in Administration > Settings > Misc.

But you should not attempt to run pip for code envs as the root user, that would most likely make the code env unusable later on, since all commands issued by DSS on the code env will be with the dataiku user. You should instead make the /directory/to/package accessible to the dataiku user, and put it in the required packages. The list of required packages accepts package names, git repositories urls, paths to tar.gz files of packages, and even paths to the folder of the source code of a package.

If using root is mandatory, then you can make a "non-managed path" code env, but then you'll have to manually setup the python virtualenv in the path of the code env, and make sure it's accessible to the dataiku user

View solution in original post

1 Reply
fchataigner2
Dataiker

Hi, 

the `pipInstallExtraOptions` is where you add command flags, such as `--proxy` and stuff, for `pip` commands. You can edit in the DSS UI in Administration > Settings > Misc.

But you should not attempt to run pip for code envs as the root user, that would most likely make the code env unusable later on, since all commands issued by DSS on the code env will be with the dataiku user. You should instead make the /directory/to/package accessible to the dataiku user, and put it in the required packages. The list of required packages accepts package names, git repositories urls, paths to tar.gz files of packages, and even paths to the folder of the source code of a package.

If using root is mandatory, then you can make a "non-managed path" code env, but then you'll have to manually setup the python virtualenv in the path of the code env, and make sure it's accessible to the dataiku user