DSS API Endpoints: Enrichments and Lookups - Variable References Question

importthepandas
importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

Hi Team

With endpoints such as lookups and for enrichments on prediction endpoints, is it expected behavior for DSS to not recognize a variable set in a connection used for the enrichment? I'm assuming this is expected based on your previous answer about python endpoints, but just want to confirm. We've a setup where we set variables to flip from "dev" to "prod" (design to auto) upon deployment for database schemas, so we use variables in the connections. We tried to use these same connections for lookup endpoints but receive errors due to the variables. We ended up working around by setting hard coded connections with an API_ prefix.

We deploy API services on EKS and set connections in the infrastructures on an independent deployer node.

Thank you!

Best Answer

Answers

  • fsergot
    fsergot Dataiker, Registered, Product Ideas Manager Posts: 118 Dataiker

    Hello @importthepandas
    ,

    To unerstand the whole context: Are you using bundled or referenced data for the lookup dataset? And for which parameter are you using the variable in the connection?

  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    @fsergot
    - thank you!

    Primarily referenced data, these are endpoints we are using to serve online features. In the connection, we are using a few variables. It's a snowflake connection we use variables for the database, write schema, table prefixes, warehouse and unqualified lookups.

  • fsergot
    fsergot Dataiker, Registered, Product Ideas Manager Posts: 118 Dataiker

    In the case of API kubernetes deployments, the referenced connections are defined in the API infrastructure screen:

    Screenshot 2022-07-19 at 08.03.08.png

    In this screen, you define all the connection details so you can actually replace any variable with its actual value in the context of this API infra.

    For example, if I am using a different schema for each node. I have a ${schema} in my connection details.

    Screenshot 2022-07-19 at 08.07.51.png

    I have an instance variable "schema" = "design" on my Design node and an instance variable "schema" = "automation" for the automation node.

    Screenshot 2022-07-19 at 08.08.33.png

    For the API cluster here, by default, I am retrieving the connection definition in my clipboard and paste it. And it contains "schemaName": "${schema}", which I can replace, in this box, by "schemaName": "api".
    Screenshot 2022-07-19 at 08.09.14.png

    Screenshot 2022-07-19 at 08.10.08.png

    Wouldn't that work for your case?

  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    @fsergot
    apologies for my delay, and thank you for your response. Indeed, hard coding references where variables once were works well and we can set dev and prod infrastructures.

    I was more curious, however, about using variables for these services both when they are being designed in the API designer, and also when they are referenced in infrastructures above as variables on the deployer. What we typically see is DSS is not picking up the variable reference, for instance when testing a lookup endpoint on the design node (attachments 1-3), and then once more on the deployer node (last attachment). I am assuming this is expected behavior as deployed services will not be aware of the variable references?

    Examples:

    1. 1. credit_feats_table_connection.png

    2. 2. endpoint_config.png

    3. 3. test_server_design_error.png

    4 (attached - couldn't edit change inline)

  • importthepandas
    importthepandas Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 115 Neuron

    thank you @fsergot
    - you guys are always on it. there is a workaround so we can make all the things work. thank you for the assist!

Setup Info
    Tags
      Help me…