Storing and Retrieving Embeddings in Knowledge Bank via Python

Hello Team,
I hope you are doing well.
I am currently working on a project in Dataiku 13.1.2, where I am generating embeddings using LLM Mesh in Python code. At present, I am storing these embeddings in a PostgreSQL dataset. However, I would like to store them directly into a Knowledge Bank using Python code.
Key Requirements:
- Store embeddings in a Knowledge Bank programmatically using Python (without using the "Embed Documents" recipe or any other recipe).
- Read data from the Knowledge Bank using Python.
Assistance Required:
- Is it possible to write/update/store embeddings into a Knowledge Bank via Python code?
- If yes, could you please share the relevant documentation link and a sample code snippet?
- How can I retrieve/read data from the Knowledge Bank using Python?
- If there is an API available, kindly provide the details and an example.
Your guidance on this would be highly appreciated. Looking forward to your response.
Regards,
Misti
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,258 Dataiker
Hi,
1) Can you write/update/store embeddings into a Knowledge Bank via Python code?Currently no, you can only retrieve text embedding, but writes must be done with an embedding recipe.
https://developer.dataiku.com/latest/concepts-and-examples/llm-mesh.html#text-embedding
https://developer.dataiku.com/latest/api-reference/python/llm-mesh.html#dataikuapi.dss.llm.DSSLLMEmbeddingsQuery
2) Yes you can use a Knowledge bank from code
https://developer.dataiku.com/latest/concepts-and-examples/llm-mesh.html#using-knowledge-banks-as-langchain-retrievers -
Thank you so much Alexandru 😊