DSS API Endpoints: Enrichments and Lookups - Variable References Question
![importthepandas](https://us.v-cdn.net/6038231/uploads/Dataiku/nAvatar11.png)
Hi Team
With endpoints such as lookups and for enrichments on prediction endpoints, is it expected behavior for DSS to not recognize a variable set in a connection used for the enrichment? I'm assuming this is expected based on your previous answer about python endpoints, but just want to confirm. We've a setup where we set variables to flip from "dev" to "prod" (design to auto) upon deployment for database schemas, so we use variables in the connections. We tried to use these same connections for lookup endpoints but receive errors due to the variables. We ended up working around by setting hard coded connections with an API_ prefix.
We deploy API services on EKS and set connections in the infrastructures on an independent deployer node.
Thank you!
Best Answer
-
Hello,
Indeed, the issue on the API Designer & Deployer is a bug and will be fixed in a coming release.
Answers
-
Hello @importthepandas
,To unerstand the whole context: Are you using bundled or referenced data for the lookup dataset? And for which parameter are you using the variable in the connection?
-
importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron
@fsergot
- thank you!Primarily referenced data, these are endpoints we are using to serve online features. In the connection, we are using a few variables. It's a snowflake connection we use variables for the database, write schema, table prefixes, warehouse and unqualified lookups.
-
In the case of API kubernetes deployments, the referenced connections are defined in the API infrastructure screen:
In this screen, you define all the connection details so you can actually replace any variable with its actual value in the context of this API infra.
For example, if I am using a different schema for each node. I have a ${schema} in my connection details.
I have an instance variable "schema" = "design" on my Design node and an instance variable "schema" = "automation" for the automation node.
For the API cluster here, by default, I am retrieving the connection definition in my clipboard and paste it. And it contains "schemaName": "${schema}", which I can replace, in this box, by "schemaName": "api".
Wouldn't that work for your case?
-
importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron
@fsergot
apologies for my delay, and thank you for your response. Indeed, hard coding references where variables once were works well and we can set dev and prod infrastructures.I was more curious, however, about using variables for these services both when they are being designed in the API designer, and also when they are referenced in infrastructures above as variables on the deployer. What we typically see is DSS is not picking up the variable reference, for instance when testing a lookup endpoint on the design node (attachments 1-3), and then once more on the deployer node (last attachment). I am assuming this is expected behavior as deployed services will not be aware of the variable references?
Examples:
1.
2.
3.
4 (attached - couldn't edit change inline)
-
importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron
thank you @fsergot
- you guys are always on it. there is a workaround so we can make all the things work. thank you for the assist!