API Services code able to produce HTTP statuses 4xx

daniel_adornes

DSS allows us to programmatically develop API endpoints logic.

The logic consists of plain Python/R code, with simple responses (primitive values or JSON) and doesn't currently offer a wait of customizing the HTTP status to of this response for consumers.

While integrating with other microservices in an enterprise environment, as a very standardized good practice, these consumers expect HTTP statuses that properly indicated what happened (ref: https://httpstatuses.com/

• 200 - Everything ran just fine
• 500 - Some internal error occurred (e.g. some error in the API developed in DSS or some unavailability of API Node)
• 4xx - A range o possible error codes that indicate specific errors as expected errors/validations which indicate that the consumer needs to fix something.

Turribeach

The Dataiku API endpoints already return valid HTTP codes. So for a successful call you will get the expected 200 (OK) code back. You can add "-i" to your curl sample code/test API command to see the HTTP status code in your test call. I think it is not possible to customise these codes.

A much more problematic issue with API endpoints is that it doesn't seem possible to throw errors if required API parameters (aka model features) are not being passed to the API endpoint. In other words an API Developer can’t set API parameters as required. This is a very basic concept in APIs where some parameters must be provided and some can be optional. In particular in ML models scoring data that lack certain parameters (aka features) may lead to wrong/invalid predictions since most ML models can't handle NULL features well.

So we can't control HTTP response codes and we can't require parameters. The final issue is that we needed to validate the API inputs to make sure they were within expected ranges. So in order to address these problems what we did is that we agreed with the calling application on a JSON format that included an error section so that we can pass any error codes we want. And we also agreed with them than any HTTP codes other 200 should result on a failure on their side. While not strictly adhering to common API HTTP codes our solution works. And to be honest I have not seen a strong standarisation around this area, some API backends return error codes on HTTP statuses, some don't. Life is hard in the world of Technology...

I still support this Idea but given that there is a work around (as above) I think Dataiku will not give it much attention. Luckily our work around allowed us to address all the issues I mentioned about but this did require a lot more code from our side to achieve it.

I do think that sometimes Dataiku spends too much resources on new features when basic features such as the ones mentioned on this Idea are not part of the product.

Hope it helps!