LLM Response streaming in API Designer
Hey, I am trying to implement a stream API in the API Designer for having the ability to request the endpoint in stream mode for displaying LLM response in a chatbot UI.
I took inspiration from Response streaming section of LLM Mesh - Dataiku Developer Guide but I am unsure what to use as a replacement of the print statement to be able to create a stream API. I tried using yield, async or flask tools but I couldn't find what to use exactly. When using yield (and eventually async def) I am getting an error "Object of type (asyc_)generator is not JSON serializable".
Basically what I want to do is to develop an API endpoint so that it can be used to progressively display the LLM response in a chatbot UI. Any guidance on the proper way to do this in Dataiku would be appreciated if the proposed approach is not the best.