Setting environment in a python plugin

davidmakovoz

I created a plugin from a python recipe. For the recipe I use my environment based on Python 3.6 with some additional packages.

When I try to use the plugin, I get the error that one of the packages cannot be imported, which I assume happens because DSS uses some default python.

I tried to create an environment for the plugin. There is a "+CREATE A CODE ENVIRONMENT" button on the Summary page of the plugin. When I click it, it opens a dialog "New Code Environment" with a drop down menu with two options: R and Python. I pick Python.

A side point, I really expected to see a dropdown menu with all the available python environments and to pick mine, but it didn't happen. Why can't I just do that?

Anyway, I pick Python, click "Add". That takes me to the Edit tab and I see that the whole new section, code-env, was added. I see desc.json file with the following content:

{
  "acceptedPythonInterpreters": [],
  "forceConda": false,
  "installCorePackages": true,
  "installJupyterSupport": false
}

I'm wondering if there is a way to add key-value pair that would tell the plugin to use my environment, something like:

"use_code_environment": my_favorite_environment

I don't see anything in the documentation or tutorials.

There is a /code-env/python/spec/requirements.txt file. I"m thinking, maybe if I add the package I need here, DSS will add the package to the default environment. I do add the package, save, reload the plugin. Try to run it again and fail again, package is not found.

Then I notice on the Summary page the following

This plugin will not work until the code environment is properly setup.

This plugin requires a dedicated code environment.

You don't have the permissions to alter the settings of the plugin.

At this point I have two questions:

1. Perhaps adding the package to the requirements.txt file didn't work because I don't have permissions to alter the settings of the plugin?

2. My profile is "Designer". Why can't I alter the settings of the plugin I created? Do I need an admin to do it for me?

Thank you,

David

tgb417

These may be a few silly questions:

Are you able to install any plugin that uses a python environment?

For example, can you install the standard google sheets plugin?

Does it create a code environment for you?

You mentioned that you were a designer. That is a "Profile", not a user group. My understanding is that Profiles are the type of license you have. If I understand correctly designer is a full license.

On user groups, there are permissions related to creating code environments and writing plugins.

If you are not in an administrator group, by default you do not get those capabilities.

I'm wondering if you are an administrator in your environment? Do you control the security on your instance?

It might help others to help sort this out if we something more about your instance are you on:

• Free vs Paid Version of DSS? (Security is different)
• What OS is running on your DSS Instance? (Linux and Mac OS may have somewhat different restrictions at the OS level.)

Marine

Hi David,

It's not possible to reuse an existing code env to run the custom recipes of a plugin. Each plugin creates a new code env thanks to the /code-env/python/spec/requirements.txt file.

You must have admin rights to create or update a code env for a plugin, which may explain why you could not set it up properly.

You also need special permissions to develop plugins within DSS. You may ask your admin to be part of the "Develop plugin" group as mentioned in Dataiku's documentation.

If you want to develop a plugin having the full admin rights, you could also develop the plugin from a community DSS instance on your local machine and then ask one of your admins to import the plugin back to your current instance.

Does this answer your questions?

tgb417

@davidmakovoz
,

Nice post. I’m not a plugin designer so, not really able to help much here.

However, I would like to invite you to contribute your idea for changing how code environments work with plugin development over on the Product Idea Forum. Getting ideas like your to the developers will make all of our lives better i the long run.

davidmakovoz

@tgb417
,

Thank you. I have contributed some ideas before.

Before making any suggestions, though, I'd like to understand better how it is supposed to work now.

David

CoreyS

Hey @davidmakovoz
you may already be past this point but while you wait for a more complete answer, just incase you haven't yet come across this resource, I wanted to share it with you from our documentation: Operations (Python) for Code Environments

I hope () this helps!

davidmakovoz

@CoreyS

Thank you for the reply. Yes, I'm definitely way past this point. I created an environment for my python recipe. Now I'm converting the recipe to a plugin. And the question is how I set the same environment for the plugin, or at least include the necessary packages in the requirements.txt file of the plugin.

Regards,

David

Ignacio_Toledo

Hi @davidmakovoz
,

I wouldn't know why you are having problems with the permissions, but I think there is something to say about the use of code environments created with DSS (in administration -> "Code envs").

I think one of the ideas of creating a plugin is not only to make a manual operation more efficient by automatizing it, but also to share this plugin with other DSS instances: if when you create the plugin you only reference an environment by using the name on your DSS instance, when you try to install it in another node or machine, a name won't give enough information about the environment it is needed to make your plugin work.

So, the two important configuration files in the plugin file directory, where you should define the environment are described at https://doc.dataiku.com/dss/latest/plugins/reference/other.html#code-environments:

• desc.json (either PYTHON27 or PYTHON36 for accepetedPythonInterpreters
• requirements.txt: for the list of packages the environment should have. Something like

pandas==1.0.5
numpy==1.19.4
pyarrow==2.0.0

Hope this helps a bit.

davidmakovoz

@Ignacio_Toledo

I agree, it makes sense not to rely on my environment, but rather define is using the requirements.txt file.

Which I did, as stated in my post. I put the package name - sentence_transformers - in the requirements.txt file and "acceptedPythonInterpreters": [PYTHON36] in desc.json file.

I get the error message

ImportError: No module named sentence_transformers

when I run the plugin.

Ignacio_Toledo

Hi @davidmakovoz

I think there is typo in the package name it should be

sentence-transformers

no underscore.

Cheers!

tgb417

@davidmakovoz
, @Ignacio_Toledo
,

I'm wondering if looking at the GitHub repo for Dataiku plugins would be helpful?

https://github.com/dataiku/dataiku-contrib

There is a set of samples here:

https://github.com/dataiku/dataiku-contrib

However, those samples do not even reflect requirements.txt and seem to be 2-6 years old.

I'm wondering about taking a look at a more mature plugin as an example might be helpful.

https://github.com/dataiku/dataiku-contrib/tree/master/googlesheets

--Tom

davidmakovoz

Yes, I had sentence-transformers in requirements.txt. I just copied sentence_transformers from the error message in the post. Sorry for the confusion. Btw, I also tried you example

pyarrow==2.0.0

and

import pyarrow

in the python file.

That also failed, couldn't import pyarrow.

davidmakovoz

@tgb417

Googlesheets has requirements.txt with the following:

pyOpenSSL
gspread==3.3.1
oauth2client==4.1.3
python-slugify==4.0.0

davidmakovoz

Hi @Marine
,

Perhaps, I accepted the answer too soon . My admin tells me that I am part of the code_enviornment custom group that has some of the advanced permissions that the Dataiku documentation refers to.

I'm rereading your answer again and now I'm not sure what you are saying.

"You must have admin rights to create or update a code env for a plugin, which may explain why you could not set it up properly.

You also need special permissions to develop plugins within DSS. You may ask your admin to be part of the "Develop plugin"

It sounds that I must have admin rights. But if I had admin rights, I wouldn't have to ask my admin anything, would I? I'd do it myself. What am I missing?

I've attached a screenshot of my permissions that my admin sent me.

Regards,

David

tgb417

@davidmakovoz
,

Although, incomplete, what you have shown here from your admin looks reasonable to me.

If @Marine
does not pop back onto this thread. You and your admin may want to put in a support ticket to get this one corrected.

You can request support by:

Marine

Hi David,

From this screenshot, it seems to me that you're not an administrator of your instance. Administrators are granted all permissions on all scopes. So if you were one, all the boxes would be checked.

I reckon the names of these permissions are confusing. Currently, neither the "code env" permissions nor the "Develop plugin" permission will allow you to create a plugin code environment. Only administrators can. I'll contact the product team to see if we can make these permissions clearer and more flexible.

davidmakovoz

Yes, the whole issue is clearly confusing. @tgb417
seems to be confused as well, and my guess we are not the only ones confused. Also, the answer you gave "You may ask your admin to be part of the "Develop plugin" is confusing, if I have the admin rights, why would I have ask anyone anything?

Btw, wouldn't it make sense to have a role that has permissions to create an environment for plugins, but not be the full blown administrator?

Marine

The screenshot that you shared shows that you don't have admin rights on your instance. Based on your previous comments, I thought earlier that you could not edit the code of the plugin. That's why I suggested that you asked your admin to add you the group who can develop plugins.

For the moment, the best practise is to develop your plugins on a local instance or on a development instance where you are an administrator. Once the plugin is completed, ask your administrator to install it and set it up on your main DSS instance.