How about tooling, what open-source licensed products are available that are easy to use both for techies and non-techies, probably one that does not need to go to Google’s servers to bring back a chart. Interactive charts would be fun, though static ones are good to start
Assuming this is related to https://wiki.openmrs.org/display/projects/Built-In+Reports+for+Reference+Application, my gut is that we might want to provide some configurable widgets/controls that can be added to the reference application, where one of the configuration options is the dataset/report to run, and other configuration options are the type of chart, and dataset elements to use, etc. But this would be something different from the typical report rendering architecture. Other widgets would just allow for configuring one or more reports to expose (eg. display name and url), which would then either run the report synchronously and download the rendered report, or redirect the user to another page…
I would use reporting REST + JS charts. There’s a bit on JS charting here.
I also would do this client-side, using some charting library that’s built for the whichever JS framework is chosen for the project.
If you’re generating numbers as a dataset in the reporting module, the reportingrest module is perfectly capable of exposing them to a JS app.
@mseaton, yes, it’s the related project, which I should have mentioned.
@lluismf, I haven’t seen Kibana before, thanks! It’s not the best fit for the project I was thinking about, because we want something simple running within OpenMRS, but it’s a great suggestion for OpenMRS data mining. It seems like a decent alternative to Pentaho. Do you have any experience with that tool? Would you be interested in setting up some basic demo for OpenMRS? I’m very interested in pairing on that.
No, I don’t have any experience but a colleague from work is evaluating it and so far he’s very happy. In a couple of days he’s loaded milions of rows in ElasticSearch (from Oracle) using LogStash with no coding (just defining SQL queries to extract the info). Then he created several graphics with Kibana (maps included because the original data has geolocation). On top of that, searching is ultra-fast thanks of Lucene magic.
I don’t have much time but yes, I would be interested but I will need some assistance. As a minimum, an OpenMRS big enough, the data to be extracted (observations?) and graphs that make most sense.
@raff Would like hear what JS library you are using for charting and experiences with it to use as a baseline for a number of planned sections on the patient and clinician facing dashboards
This is a very exciting development. Will this be designed to be distribution agnostic, or is this intended for the Ref App?
@lluismf, let’s catch up early next week.
So I guess you would first provide a new REST API, some sort of
ChartResource? … that would hold series of plottable data or tables of non-plottable data that could easily be consumed by front-end libraries?
@mksd, the plan is to use https://wiki.openmrs.org/display/docs/Reporting+REST+Web+Services to feed the charts.
@mseaton am wondering how far this has gone? we need to implement something similar in UgandaEMR, that is, displaying charts on a dashboard widget using data from the reporting module.
@solemabrothers, many sites have moved from this approach to an approach of flattening data within a secondary datastore and performing analytics there. Some have created bespoke SQL queries to do the flattening (can require some large & hard to maintain queries) and, more recently, people have been looking to existing open source tooling (e.g., debezium and Apache tools) to flatten data. There have been successes, but many are implementation-specific solutions.
PIH created petl as an extensible approach to defining flattening & reporting queries in SQL in a standard way that can be run by an engine.
The Analytics Engine Squad was created by groups trying to solve this problem in a way that could benefit others and be extended by the community. AMPATH and Google built openmrs-fhir-analytics to extract data into a FHIR schema and use Apache parquet (columnar datastore) to tackle the calculation of indicators.
@mseaton, @allan, @bashir, and myself have been trying to move our community reporting discussions in 2022 toward sharing approaches across four areas: CDC, ETL, data store, and analytics. For example, most approaches have found debezium to be the best tool for CDC.
We’d love to get folks from UgandaEMR, NigeriaMRS, KenyaEMR, PIH, AMPATH, Mekom, etc. (all of whom are trying to solve this problem in different ways) to share approaches & coordinate efforts on reporting & analytics to reduce unnecessary duplication of efforts, increase awareness & adoption of new tools (debezium, Apache tools, etc.), and (hopefully) begin discovering the parts of reporting that can benefit from shared tooling in place of bespoke solutions.
@burke thank you for the detailed response, as UgandaEMR, we have been majorly focussed on the utilization of the Reporting module to generate all forms reports that are extracted as excel files, csv and recently JSON.
However, we need to add an extra component of analytics and visualization from the the already existing data.
I am therefore sure that as a team we will benefit from the measures being undertaken here and we are going to look at the available solutions suggested especially one by PIH petl to see what lessons we can pick.
Otherwise as a team we are ready to start this journey and kindly let us know in which way we can be part of this initiative.
Great to hear! I think @mseaton is going to post something in the near future, building on our recent Reporting Strategy for 2022 discussions, to whet the appetite of your team and others who are trying to build solutions in the reporting space. I’m hopeful we can find some shared problems and willingness to work on some shared approaches/tooling.
@burke Thanks, unfortunately we need this sooner than later and built within OpenMRS distro rather than needing external tools, so we are going to figure out our own way to get it done to meet our deliverables
Understood @ssmusoke. That’s often how life goes. You’re not alone.
Even if you need to build a bespoke solution to meet immediate needs, I hope you and your team can still participate in community discussions to evolve OpenMRS to include a scalable analytics solution out of the box that can be customized to meet your needs. We want to make sure your voice is heard!
I think a group in Nigeria created a module on the NMRS for this, I saw their charts and various reports.