Finding a way forward for OpenMRS reporting

While we’ve made progress on analytics over the past year, it’s been almost exclusively an effort between AMPATH and Google piloting a scalable approach toward generating indicators and we’ve failed to get widespread engagement on a shared approach for reporting. The community-friendly approach of openmrs-fhir-analytics, leveraging standards, our squad model, and best practice tooling, has been top notch. The lack of adoption, I believe, is most OpenMRS implementing organizations lack the expertise to directly benefit/engage.

I still believe in the potential for reporting as a micro service of the platform via a dockerized “data warehouse” that not only pulls data out of the EMR, but also provides data back to the EMR as derived concepts and aggregate knowledge. I also believe we’re going to be best served leveraging open source tools (Apache, etc.) to do the bulk of the work, allowing the OpenMRS community to focus on the rules/knowledge rather than the reporting framework itself, is the best longterm solution; however, getting such a framework in place will take time and expertise.

In the meantime, we need an approach for reporting that is within reach of existing implementations. There is growing interest and pressing need for such a solution. For example, OHRI and OpenMRS 3 need a data analytics solution to drive reporting, decision support, etc.

While there isn’t single reporting solution that will meet everyone’s needs (reporting needs are largely driven by local needs & the ability to understand how to coerce local data to the needed knowledge), I do believe we can do better than leaving every implementation or organization to build their reporting solutions from scratch. While we’ve gotten a ton of benefit from the OpenMRS reporting framework, we know implementations need a pathway to flatten data outside of OpenMRS.

For example, could we define a dockerized “data warehouse” container that could serve as the destination for tools like PIH’s PETL or @aojwang’s auto-generated flattened tables? It might be used by openmrs-fhir-analytics if a SQL-based sink was needed.

I’d hope to hear from folks for a technical discussion to weigh our options. At a minimum, I’m looking toward:

  • @mseaton – long history of reporting-based work within OpenMRS
  • @eudson – to represent OHRI requirements
  • @jdick – to represent OpenMRS 3 requirements
  • @akimaina – to represent AMPATH requirements and well-positioned to provide advice on how such an effort could complement existing analytics engine work
  • @mksd – for a Mekom perspective
  • @aojwang – for KenyaEMR perspective

Ultimately, I’d hope approaches like openmrs-fhir-analytics would make the need to manually flatten data unnecessary, but until that day, is it useful to imagine “standardizing” around a data warehouse container approach (presumably MySQL… maybe MariaDB) against which we could to share basic tooling (e.g., debezium, PIH’s PETL, etc.) so every organization doesn’t have to start from scratch?

1 Like

I totally agree on the need for a comprehensive and out of the box flattening mechanism for the data in OpenMRS. From a user perspective, I expect the reporting platform to address the following:

  1. Generation of standard reports i.e. the PEPFAR indicators and the MOH reports
  2. Ad-hoc data requests to help drive data use at facility and service delivery partner level. This should take into account what the different users at different levels would want to see in reporting datasets i.e. the column names and values.
  3. Faster process of building and rebuilding of the flat tables. Debezium based approach sounds appropriate for incremental updates
  4. A mechanism that scales well both for low and high volume data, and resource constrained settings
  5. An isolated process for managing all reporting processes as this will help shield the EMR from the effects of long running reports which eventually slows down the system. My hope, however, is to have all reports generated in seconds if not a few minutes.
  6. Ability to monitor and manage the reporting micro-service from the EMR?

Looking forward to other responses. Cheers!