How to help with O3 performance?

Hello Community,

At SolDevelo, we’re keen to help tackle some of the performance challenges in OpenMRS O3. We want to focus on real issues that users face—ones where improvements would bring meaningful value to the community. But to make sure we’re addressing the right problems, we’d love some guidance!

I came across some discussions on Talk, with the latest one being: O3 Performance Improvement And the related Epic: O3-2621 However, both are a few months old, so I’m not sure how up-to-date they are.

From what I’ve gathered, there seem to be two possible areas where we could contribute:

  1. Fixing specific known bottlenecks – for example, adding pagination to certain endpoints that currently slow things down.
  2. Reproducing real-world deployment scenarios – setting up an environment to simulate actual performance issues that users encounter.

The first option seems like a quicker win to get started and make an immediate impact.

What do you think? Are there specific areas where our efforts would be most valuable?

Looking forward to your thoughts!

cc: @jslawinski @druchniewicz

6 Likes

@grace @burke are we able to gather the volumes of data that OpenMRS should support? What are the biggest instances currently deployed?

What about findings from Optimizing the Backend: Examples of slow responses in O3?

1 Like

@ibacher did we gain any significant performance improvements in your attempt at loading of locations here? FM2-642: Improve performance of loading locations by tag by ibacher · Pull Request #547 · openmrs/openmrs-module-fhir2 · GitHub

If not, would this be a good one to add to the SolDevelo menu?

1 Like

Weird one… I was able to reproduce a performance bug which that PR fixed, but it didn’t seem to have as dramatic of an impact on overall end-point performance, so yes, it’s still there, but I don’t know what approach to recommend.

That said, we have a report for the daily tests here: https://o3-performance.openmrs.org/ and you can see what queries are slow from that. The underlying code it here: openmrs-contrib-performance-test/src/test at main · openmrs/openmrs-contrib-performance-test · GitHub, which explains what those labels correspond to.

A simpler place we need help is simply in adding more things to be tested. We have just two personas who simulate two flows: registration and filling in a visit form, so building out more scenarios would likely be an even bigger win if it helps us highlight other places in the O3 application where data-fetching takes longer than it should.

3 Likes

It looks like the testing scenarios are planned to be covered by the Performance Testing Enhancement GSoC 2025 Project.

Looking at the performance stats the Get Locations indeed looks like the hottest endpoint and an interesting challenge.

Edited 2025-03-10: We’ll have a look at the locations endpoint - maybe a new set of eyes would be the key here.

1 Like

Hi everyone! :waving_hand: I’m currently working on the Performance Testing Enhancement Project for OpenMRS as part of GSoC 2025. I’ve been setting up the testing framework and working on adding new personas (Clerk, Nurse, Doctor) to simulate real-world workflows.

I noticed the discussion about performance bottlenecks, especially with the Get Locations endpoint. If it’s a known issue, would adding specific load tests for it be useful? I’d love to contribute by reproducing the issue within our performance tests and seeing how it behaves under different conditions.

Let me know if this aligns with current priorities or if there are other areas where more testing would be valuable. :rocket:

Hi @deepakjain, I think @ibacher or @dkayiwa can answer it best.

I’m going to look into the location endpoint, so from my perspective getting a repeatable tests made would be useful to easily judge the improvement - but my perspective and the OMRS might be different in this case. :slight_smile:

1 Like

Those performance tests are actually run daily, so they’re very repeatable.

@pwargulak does it mean we have everything to verify the fix?

@jslawinski we do have all.

@ibacher It’s clear.

Hello,

we’ve looked into the performance of the FHIR Get Location endpoint, and we found some interesting solution that improves the endpoint by 70-85%, but it does involve some design change around FHIR Translator classes.

Notes on the results

  • it was measured using the openmrs-contrib-performance-test project, export SIMULATION_PRESET=dev TIER_COUNT=2 TIER_DURATION_MINUTES=1 && ./mvnw gatling:test
  • it was executed on a Local as-is
  • and it was executed with an artificial network Delay of 5ms, added at docker container level, to all containers, mainly to highlight an impact of DB queries and simulate deployment in a cloud (slow cloud :slight_smile: )

The results

[ms] Local - 95th percentile Local - Mean Delay - 95th percentile Delay - Mean
Before 271 200 3762 3555
After 58 35 1361 955
Gain +78% +82% +63% +73%

See ZIP file attached to O3-4544 for full results.

The change

I’ve made a PR with the code-change here. Some technical details are included in the PR’s description - but the short explanation is that we reduced the number of DB queries.

The commit is named PoC, as it affects only the Get Location and the idea/implementation might need further refinement. This change just seemed to have the least effort with a nice impact.

The comments

The PR itself shouldn’t break anything, but since it introduces a change in very common API, it would be great to hear what do Community thinks.

I’m not sure if you get updates about this thread so I’m pinging @dkayiwa @ibacher.

6 Likes

To the @SolDevelo team, this is super awesome!!! I tested the performance gains in a local instance of O3 and they were very significant!!! :smile:

Now iam diving into the code review and evaluation of your approach.

3 Likes

Thanks, @dkayiwa! @ibacher just merged it today.

We’re really happy to contribute, and it’s great to see the impact it had!

Lately, at SolDevelo, we’ve been focusing a lot on Java performance, and we’d love to bring similar improvements to a wider audience. It would really help us if we could share a short testimonial on our website to reinforce the promise that we can deliver these kinds of optimizations. :slight_smile:

Maybe @grace could help with that?

1 Like