The specified Code Environment is not used when running tests for Python Function API Endpoint
I have setup a Conda environment which I am successfully using for two Webapps and some recipes.
However, when trying to create a Python Function API Endpoint, and using the "RUN TEST QUERIES" function, I get errors such as "ModuleNotFoundError: No module named 'transformers'" indicating that the prepared Code Environment is not used. Also, when observing the list of virtual environments in Anaconda, I can see that new VMs are created every time I run test queries, with the general name "env" and not using the dedicated "api_p38_env". The environments called "env" have completely different packages than the environment I have prepared.
I have specified theCode Environment settings as can be seen in the attached screenshot.
Also, I tried using two different Code Environments, so that the one used for the two Webapps is separate, but to no avail.
The log file for the test, including errors, has been attached. I would appreciate any help.
I noticed that my issue looks similar to this one: https://community.dataiku.com/t5/Using-Dataiku/Use-file-from-code-env-resources-directory-with-API-Test-queries/m-p/31771/highlight/true#M11785
Operating system used: MacOS 13.3.1 (a), Arm64
Operating system used: MacOS 13.3.1 (a), Arm64
Best Answer
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 DataikerHi @Wolfram36
,I see, real-time API's are not listed under the free license version. Sorry about that. You may want to explore the Dataiku Online 14 day free trial: https://www.dataiku.com/product/get-started/
Thanks!
Jordan
Answers
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 Dataiker
Hi @Wolfram36
,The transformers package was likely not there at the time that the dev server started and installed your code env. To make sure that the transformers package gets installed, please complete the following steps:
Double-check that transformers is listed in the "packages to install" tab under "requested packages". Then, please rebuild your code environment from the Administration > Code envs with the "Rebuild env" checkbox checked:
Next, please restart your API dev server from Administration > Settings > Deployer:
And, test your api test query once more.
Please send us an updated log if you receive an error.
Thanks!
Jordan
-
Thanks @JordanB
for your fast assistance.I carried out the steps you suggested. (Initially, I used Terminal and pip install to setup the virtual environment on Anaconda, but this time I added all packages on Dataiku instead.)
Still, when starting the test, I can see that a new virtual environment called "env" is launched in Anaconda. But, it seems that this new virtual environment has the packages as declared in Dataiku.
The new error is:
com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to initiate function server : <class 'Exception'> : Default project key is not specified (no DKU_CURRENT_PROJECT_KEY in env)
See attached log file and screenshot.
I referred to the following instructions, but could not figure out how to specify the DKU_CURRENT_PROJECT_KEY and how to use it in my API:
- https://community.dataiku.com/t5/Using-Dataiku/How-to-reference-global-variables-in-API/m-p/8319
- https://community.dataiku.com/t5/Plugins-Extending-Dataiku/Uploading-a-file-to-a-folder-using-the-Python-API/m-p/10554
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 Dataiker
Hi @Wolfram36
,The error message is telling us that it does not know to which project should it connect.When running the same code from a notebook you'd not hit this error because 'DKU_CURRENT_PROJECT_KEY' corresponds to the project in which the notebook exists.The code the API designer is running has no notion by default of what project is supposed to be talking to. Furthermore, the idea is that ultimately the code will be run from the API node, which is outside of the Design node.You will need to install the dataikuapi version of the dataiku package and set_remote_dss(). You will have to connect to DSS instance by host and api-key, then connect to the project by the project key and then you can run the rest of your code.import dataikuapi # Connect to DSS and get a client dataiku.set_remote_dss("http://localhost:port", "insert-api-key") client = dataiku.api_client()https://doc.dataiku.com/dss/latest/python-api/client.html#reference-api-doc
https://doc.dataiku.com/dss/latest/python-api/outside-usage.htmlPlease let us know if you run into any further issues.Thanks!Jordan -
Hi @JordanB
I followed the following steps:
- Added 'dataiku-api-client' to the list of packages to install for my selected virtual environment
- Next, I wanted to create an API key, so I navigated to 'http://localhost:11200/projects/myprojectname/security/api' but I was met with the text "The public API is not available in this version of DSS."
- Next attempt was to navigate to 'http://localhost:11200/projects/myprojectname/api-designer/my-api/config/'. This screen allows creation of API Keys, but after refreshing the page, the key is gone, so I am not sure how to revoke an issued key in the future.
- Anyways, now I had an API key, so I added the following code on top of the code for my API:
import dataiku import dataikuapi import logging host = "http://localhost:11200" apiKey = "my--newly--created--key" client = dataikuapi.DSSClient(host, apiKey) logging.info(client.list_project_keys())
But, now I get the error "Failed to initiate function server : <class 'dataikuapi.utils.DataikuException'> : com.dataiku.dip.exceptions.NotAuthenticatedException: Unknown API Key"
Are my difficulties related to my license (i.e. the Free version)? I am still evaluating your product and could potentially switch to sign up for a limited trial, if that helps.
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 297 Dataiker
Hi @Wolfram36
,You need a global (instance) API key for this, not a project level api key. What happens when you select your icon at the top right > profile and settings > API Keys > +New API Keys.
Can you please give that a try?
Thanks!
Jordan
-
Hi @JordanB
I have attached a screenshot for what I see after clicking on that link.
It seems that I cannot create an API for a Python function with the FREE license.
-
I feel a bit uneasy putting my global api key into code that could be visible to others collaborating on the project. Is there a better way to do this? Otherwise, someone can take my api key and get access to things they may not otherwise have, or do other things while impersonating me.
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,188 NeuronThis thread is over a year old and already marked as answered. Please start a new thread.