Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

DSS API Endpoints: Enrichments and Lookups - Variable References Question

Solved!
importthepandas
Level 4
DSS API Endpoints: Enrichments and Lookups - Variable References Question

Hi Team

With endpoints such as lookups and for enrichments on prediction endpoints, is it expected behavior for DSS to not recognize a variable set in a connection used for the enrichment? I'm assuming this is expected based on your previous answer about python endpoints, but just want to confirm. We've a setup where we set variables to flip from "dev" to "prod" (design to auto) upon deployment for database schemas, so we use variables in the connections. We tried to use these same connections for lookup endpoints but receive errors due to the variables. We ended up working around by setting hard coded connections with an API_ prefix.

We deploy API services on EKS and set connections in the infrastructures on an independent deployer node.

Thank you!

0 Kudos
1 Solution
fsergot
Dataiker
Dataiker

Hello,

Indeed, the issue on the API Designer & Deployer is a bug and will be fixed in a coming release.

View solution in original post

0 Kudos
6 Replies
fsergot
Dataiker
Dataiker

Hello @importthepandas ,

To unerstand the whole context: Are you using bundled or referenced data for the lookup dataset? And for which parameter are you using the variable in the connection?

0 Kudos
importthepandas
Level 4
Author

@fsergot  - thank you!

Primarily referenced data, these are endpoints we are using to serve online features. In the connection, we are using a few variables. It's a snowflake connection we use variables for the database, write schema, table prefixes, warehouse and unqualified lookups. 

0 Kudos
fsergot
Dataiker
Dataiker

In the case of API kubernetes deployments, the referenced connections are defined in the API infrastructure screen:

Screenshot 2022-07-19 at 08.03.08.png

In this screen, you define all the connection details so you can actually replace any variable with its actual value in the context of this API infra.

For example, if I am using a different schema for each node. I have a ${schema} in my connection details.

Screenshot 2022-07-19 at 08.07.51.png

 

I have an instance variable "schema" = "design" on my Design node and an instance variable "schema" = "automation" for the automation node.

Screenshot 2022-07-19 at 08.08.33.png

For the API cluster here, by default, I am retrieving the connection definition in my clipboard and paste it. And it contains  "schemaName": "${schema}", which I can replace, in this box, by "schemaName": "api".
Screenshot 2022-07-19 at 08.09.14.png

 

Screenshot 2022-07-19 at 08.10.08.png

Wouldn't that work for your case?

0 Kudos
importthepandas
Level 4
Author

@fsergot apologies for my delay, and thank you for your response. Indeed, hard coding references where variables once were works well and we can set dev and prod infrastructures.

I was more curious, however, about using variables for these services both when they are being designed in the API designer, and also when they are referenced in infrastructures above as variables on the deployer. What we typically see is DSS is not picking up the variable reference, for instance when testing a lookup endpoint on the design node (attachments 1-3), and then once more on the deployer node (last attachment). I am assuming this is expected behavior as deployed services will not be aware of the variable references?

 

Examples:

1. 1. credit_feats_table_connection.png

2. 2. endpoint_config.png

3. 3. test_server_design_error.png

 

4 (attached - couldn't edit change inline)

 

 

0 Kudos
fsergot
Dataiker
Dataiker

Hello,

Indeed, the issue on the API Designer & Deployer is a bug and will be fixed in a coming release.

0 Kudos
importthepandas
Level 4
Author

thank you @fsergot - you guys are always on it. there is a workaround so we can make all the things work. thank you for the assist!

0 Kudos