The specified Code Environment is not used when running tests for Python Function API Endpoint

Wolfram36
Wolfram36 Registered Posts: 4 ✭
in Setup & Configuration

I have setup a Conda environment which I am successfully using for two Webapps and some recipes.

However, when trying to create a Python Function API Endpoint, and using the "RUN TEST QUERIES" function, I get errors such as "ModuleNotFoundError: No module named 'transformers'" indicating that the prepared Code Environment is not used. Also, when observing the list of virtual environments in Anaconda, I can see that new VMs are created every time I run test queries, with the general name "env" and not using the dedicated "api_p38_env". The environments called "env" have completely different packages than the environment I have prepared.

I have specified theCode Environment settings as can be seen in the attached screenshot.

Also, I tried using two different Code Environments, so that the one used for the two Webapps is separate, but to no avail. 

The log file for the test, including errors, has been attached. I would appreciate any help.

I noticed that my issue looks similar to this one: https://community.dataiku.com/t5/Using-Dataiku/Use-file-from-code-env-resources-directory-with-API-Test-queries/m-p/31771/highlight/true#M11785

setting.png

 

code_env.png

Operating system used: MacOS 13.3.1 (a), Arm64

Operating system used: MacOS 13.3.1 (a), Arm64

Tagged:
· Share on FacebookShare on Twitter

Best Answer

Answers

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 295 Dataiker

    Hi @Wolfram36
    ,

    The transformers package was likely not there at the time that the dev server started and installed your code env. To make sure that the transformers package gets installed, please complete the following steps:

    Double-check that transformers is listed in the "packages to install" tab under "requested packages". Then, please rebuild your code environment from the Administration > Code envs with the "Rebuild env" checkbox checked:

    Screen Shot 2023-05-17 at 5.22.00 PM.png

    Next, please restart your API dev server from Administration > Settings > Deployer:

    Screen Shot 2023-05-17 at 5.43.30 PM.png

    And, test your api test query once more.

    Please send us an updated log if you receive an error.

    Thanks!

    Jordan

    · Share on FacebookShare on Twitter
  • Wolfram36
    Wolfram36 Registered Posts: 4 ✭

    Thanks @JordanB
    for your fast assistance.

    I carried out the steps you suggested. (Initially, I used Terminal and pip install to setup the virtual environment on Anaconda, but this time I added all packages on Dataiku instead.)

    Still, when starting the test, I can see that a new virtual environment called "env" is launched in Anaconda. But, it seems that this new virtual environment has the packages as declared in Dataiku.

    The new error is:

    com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to initiate function server : <class 'Exception'> : Default project key is not specified (no DKU_CURRENT_PROJECT_KEY in env)

    See attached log file and screenshot.

    anaconda_newly_created_env.png

    I referred to the following instructions, but could not figure out how to specify the DKU_CURRENT_PROJECT_KEY and how to use it in my API:

     

    · Share on FacebookShare on Twitter
  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 295 Dataiker
    edited July 17

    Hi @Wolfram36
    ,

    The error message is telling us that it does not know to which project should it connect.
    When running the same code from a notebook you'd not hit this error because 'DKU_CURRENT_PROJECT_KEY' corresponds to the project in which the notebook exists.
    The code the API designer is running has no notion by default of what project is supposed to be talking to. Furthermore, the idea is that ultimately the code will be run from the API node, which is outside of the Design node.
    You will need to install the dataikuapi version of the dataiku package and set_remote_dss(). You will have to connect to DSS instance by host and api-key, then connect to the project by the project key and then you can run the rest of your code. 
    import dataikuapi
# Connect to DSS and get a client
dataiku.set_remote_dss("http://localhost:port", "insert-api-key")
client = dataiku.api_client()
    Please let us know if you run into any further issues.
    Thanks!
    Jordan
    · Share on FacebookShare on Twitter
  • Wolfram36
    Wolfram36 Registered Posts: 4 ✭
    edited July 17

    Hi @JordanB

    I followed the following steps:

    1. Added 'dataiku-api-client' to the list of packages to install for my selected virtual environment
    2. Next, I wanted to create an API key, so I navigated to 'http://localhost:11200/projects/myprojectname/security/api' but I was met with the text "The public API is not available in this version of DSS."
    3. Next attempt was to navigate to 'http://localhost:11200/projects/myprojectname/api-designer/my-api/config/'. This screen allows creation of API Keys, but after refreshing the page, the key is gone, so I am not sure how to revoke an issued key in the future.
    4. Anyways, now I had an API key, so I added the following code on top of the code for my API:

    import dataiku
import dataikuapi
import logging
host = "http://localhost:11200"
apiKey = "my--newly--created--key"
client = dataikuapi.DSSClient(host, apiKey)
logging.info(client.list_project_keys())

    But, now I get the error "Failed to initiate function server : <class 'dataikuapi.utils.DataikuException'> : com.dataiku.dip.exceptions.NotAuthenticatedException: Unknown API Key"

    Are my difficulties related to my license (i.e. the Free version)? I am still evaluating your product and could potentially switch to sign up for a limited trial, if that helps.

    · Share on FacebookShare on Twitter
  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 295 Dataiker

    Hi @Wolfram36
    ,

    You need a global (instance) API key for this, not a project level api key. What happens when you select your icon at the top right > profile and settings > API Keys > +New API Keys.

    Screen Shot 2023-05-19 at 10.46.40 AM.png

    Screen Shot 2023-05-19 at 10.47.30 AM.png

    Can you please give that a try?

    Thanks!

    Jordan

    · Share on FacebookShare on Twitter
  • Wolfram36
    Wolfram36 Registered Posts: 4 ✭

    Hi @JordanB

    I have attached a screenshot for what I see after clicking on that link.

    It seems that I cannot create an API for a Python function with the FREE license.

    · Share on FacebookShare on Twitter
  • mccroden
    mccroden Registered Posts: 1 ✭✭✭
    https://community.dataiku.com/discussion/comment/34452#Comment_34452

    I feel a bit uneasy putting my global api key into code that could be visible to others collaborating on the project. Is there a better way to do this? Otherwise, someone can take my api key and get access to things they may not otherwise have, or do other things while impersonating me.

    · Share on FacebookShare on Twitter
  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,925 Neuron
    https://community.dataiku.com/discussion/comment/44478#Comment_44478

    This thread is over a year old and already marked as answered. Please start a new thread.

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…