How can a Python function endpoint (API Designer) read project variables / secrets, or what's the ri

William Dieter

Goal: My API service returns document-classification predictions. I also want the same service (or a companion endpoint) to persist reviewer feedback so I can retrain later — ideally into a dataset, or into an external store (a SharePoint list via the Microsoft Graph API).

What I've found / problem:

• Inside the Python function endpoint, dataiku.get_custom_variables() returns {}, so project variables don't resolve — I can't read my SharePoint app credentials (tenant/client id/secret) that I stored as project variables.
• dataiku.api_client() on the dev server points at the sealed local backend, and calling the management public API (e.g. listing managed folders) returns 404, so I can't write to a managed folder or dataset from the endpoint either.
• The only things that reliably reach the endpoint are the request payload and files in bundled managed folders.

Questions:

1. What is the recommended way to supply secrets/configuration (API credentials) to a Python function endpoint without putting them in the request payload? Are project/user secrets, code-env resources, or bundle-included files supported and resolvable at scoring time on an API node?
2. What is the supported pattern to write/persist data (a dataset row, or an external API call) from an API Designer endpoint? Is this discouraged, and should feedback persistence instead live in a webapp backend or a scenario with full project context?
3. If a webapp backend is the recommended approach, can it read project variables / user secrets and write to a dataset directly, and is there a clean way to call it alongside my scoring API?

Dataiku version used: DSS 14.2.1

Turribeach

The API node hos no concept of projects nor can it access any project data like project variables or managed folders (aside from those bundled for models). You can however obtain this from your Designer or Automation node using the Python API (see Connecting to your Dataiku instance). However this is an undesirable pattern, as your API node will in effect depend on your Designer or Automation node. APIs are meant to be stateless and self-contained.

1. See above.

2. You can the Python API to write back to a dataset but again this is a bad pattern as it adds a dependency and will down your API calls. Look at either using the built-in API node logging or develop your own local file logging in your Python function which will be much faster and will not depend on external infrastructure so it will nearly never fail. Then copy and load these log files in your Designer or Automation node to get the desired data.

3. Yes, webapps will have automatic access to project variables / user secrets as they run inside a project and user a user identity. Notably Dataiku Webapps now support also working as API endpoint so this may suit your use case better. Although these run on the Designer or Automation node so they don't support high availability.