[BUG] API Service endpoint with enrichment through SQL connection
![daniel_adornes](https://us.v-cdn.net/6038231/uploads/lithium_attachments/n3632i4F9629B278F3A915.jpg)
Hi everyone!
So, this is a solved issue that might be helpful for someone else.
I'm using DSS 9.0 and this was my experience with endpoint enrichment through SQL connection.
When configuring a prediction endpoint to enrich the incoming request with new features, there are two options:
1) Bundle the enrichment dataset together with the model (basically copying it fully - not optimal) or
2) Configuring it to execute an on-line query to a SQL database while serving the model (probably best option in most scenarios).
When trying to use the second option, I first forgot to set the corresponding connection in the infrastructure configuration of my Kubernetes cluster in the DSS Deployer. The container failed to start up on Kubernetes accusing connection not found (easy one). Then, I set the correct connection by using the dropdown list, which automatically copied the corresponding json configuration to my clipboard. Then I just needed to add it as a connection below (see attachment).
The buggy part is that DSS stores the password in an encrypted format. So, this json copied to the clipboard (to make your life easier) will actually fail when serving the requests. The SQL query will try the encrypted password as if it was the real password for the user of your database. The solution is to replace the encrypted password by the plain password in the json configuration. Don't forget that you need to redeploy your API Service.
One other minor issue (maybe a bug, maybe not), is that when configuring a SQL connection in the global administration settings, there is the option of setting the full URL connection. If you set that, even partially (not right) and change your mind and set then the distinct fields (for host, user, password, database, etc.), DSS will still store that "dirty" value (under ...DATA_DIR/configs/connections.json). Then, when you are doing what I described before (setting a connection for an endpoint to enrich request features), the retrieved json (that one aimed at making your life easier) will bring the dirty value, which may cause confusion, though it has absolutely no effect as soon as useURL is set to false.
That's it! Enjoying the ride and wishing Dataiku gets better and better! Also, hopefully it is helpful to someone else.
Answers
-
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
Welcome to the Dataiku Community.
And thank you for this wonderful post. I love all of the detail you have shared here.
cc: @CoreyS
-
daniel_adornes Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 30 ✭✭✭✭✭
Thks @tgb417
!! Appreciate that! Have a great week!