Thanks for inquiring @grace for starting this. after successfully demonstrating the text-to-sql agent at @EMR4All we developed a few months ago, next in the pipeline is Report generation AI, These are the observations from our experience for implementing the text-to-sql agent
- Good choice of LLM is paramount, we tried out qwen2.5 , deepseek-r1:1.5b. These performed better while doing inference for online hosted models via smollagents
- They perform better while doing inference on models hosted by third parties like OpenAI, Hugging face and other vendors. We experimented with Hugging face smolagents
- We have experimented with the Offline text-to-sql agent (i.e inference on ollama powered LLM’s Qwen, deepseek hosted on PC and Raspberry PI ) this is for those who care about data privacy and want to keep their transactions completely on-prem. Results where satisfactory, though greater improvements for accuracy and alot of refining is still a work in progress.
I strongly agree, we want to standardize QA practices for the various LLM’s , frameworks and agents we’re using tools like Deepeval or any other that is at our disposal
We’d be happy to share the work we have so far to OpenMRS, and collaborate with those specifically interested in working on AI generated reports. Next Steps setting up standards, tools and best practices for implementing AI with OpenMRS, documentation, security, human in loop, agents, tools and patterns. e.t.c
Since we’ve also been looking at other practical use cases, for example the ones you highlighted above, we’d be happy to collaborate with PIH /others and come up with those MVP’s to demonstrate
@grace i’d suggest that you organize a brainstorming meeting on how to move this forward. We could especially take advantage of the forth coming Hackathon Calling for mentors and peer reviewers to join the EMR4All Hackathon.
cc @bennyange