OpenMRS starts faster than ever!

1

I’m super excited to summarise the work done in TRUNK-6417. Let me thank our volunteer contributors first: @iam0mujuzi and @suubi7! Thanks to @mseaton and @ibacher for module fixes and @dkayiwa for code reviews and testing. Finally thanks to @OpenMRSInc for finding a way to fund my work on this!

All the work is available in openmrs-core and platform 2.9.x. Some improvements have also been done in modules.

As reported by our new StartupPerformanceIT tests Platform 2.9.x starts up ~24% faster compared to Platform 2.8.x. Also it’s worth noting that StartupPerformanceIT is limited to 2 vCPUs to simulate lesser hardware. On machines with more cores, the improvements are even more significant thanks to parallel processing.

Similar results (~18%) are observed for O3 running on openmrs-core 2.9.x, which is available for testing on dockerhub in the 3.6.x-core-2.9 tag.

O3 on my machine initially started up in 45.8 seconds (as indicated in this thread). It now starts in 18.5 seconds!

For the full list of changes please see TRUNK-6417.

To get some benefits without upgrading to 2.9.x:

  1. Upgrade from JDK8 to JDK11
  2. Upgrade to webservices.rest 3.1.0-SNAPSHOT
  3. Upgrade to attachments 4.0.0-SNAPSHOT
  4. Upgrade to event 4.0.0-SNAPSHOT

If you upgrade to core 2.9.x, you will benefit from additional optimisations:

  1. Fixed to redeploy war only upon changes
  2. Fixed to unpack modules only upon changes instead of on each restart
  3. Skipped db schema checks and initial setup if version of core or module is same

Also it’s worth noting that since OpenMRS Core 2.9.x we are issuing a warning, if we detect a very common issue with OpenMRS services declared in xml using @Autowired, which may slow down each startup a few times. See TRUNK-6421. It should help distributions to quickly identify this particular startup performance issue within their custom modules.

I’ll be joining the platform call next week to go through changes and answer any questions.

Please let us know what your results are! To compare O3 startup clone openmrs-distro-referenceapplication and run:

TAG=3.5.0 docker compose -f docker-compose.yml  up
TAG=3.6.x-core-2.9 docker compose -f docker-compose.yml up

Initial setup/upgrade (first start) is long (~10 minutes, we haven’t worked on improving that yet) so test consecutive starts i.e. re-run these commands after initial setup is done.

15 Likes
OpenMRS startup time analysis
2

Great work thanks @raff / @iam0mujuzi / @suubi7

1 Like
3

Excellecnt @raff

1 Like
4

This is amazing @raff - looking forward to testing it out! Thanks!

5

@raff should i expect the same percentage improvement when running O3 without using docker? That is, when using something like the SDK? I am running O3 on openmrs-core 2.8.x with Java 21 and then comparing it with O3 on openmrs-core 2.9.x still with Java 21

6

The only thing that is Docker specific is TRUNK-6508. Basically implementers should make sure that if there’s any startup script that moves war around before starting up Tomcat, the last modified date of war is kept so that Tomcat doesn’t redeploy the war, if it doesn’t change. I’m not sure if it’s handled properly by SDK… Worth checking if there’s any entry in Tomcat logs upon each start that the war changed and it is going to be re-deployed.

Docker deployments are much easier to work with when optimising performance, because there’s less variables involved that different environments might have.

If you run with SDK, one thing you need to make sure is that it is using newly downloaded artifacts from our repos and not cached ones for war and modules. Adding “-U” when fetching artifacts should suffice. What are your results with SDK?

EDIT: TRUNK-6509 might also be affected, if omods are copied over without keeping last modified date…

7

I’ve just tried setting up O3 with SDK choosing 3.6.0-SNAPSHOT and it is actually running on openmrs-core-2.9x, which is wrong… Something is misconfigured in our O3 build…

8

Strange because i have just rerun the SDK setup now and i still get a openmrs-2.8.2-SNAPSHOT.war file.

9

Fixed it an hour ago :slight_smile:

BTW how do you go around SDK-395 ?

10

I run the SDK with -DjvmArgs="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED"

11

I’m pretty sure that moving the WAR and OMODs was a “innovation” in the Docker setup. The SDK just serves them in-place.

1 Like
12

Thanks so much @raff and all who worked on this, looking forward to testing!

13

Thanks @raff :clap:

14

@raff do you mind sharing your performance findings when running this with the SDK?

15

@dkayiwa :

O3-core-2.8

2:23:44, 2:24:30 - 1m 14s

2:25:24, 2:26:09 - 45s

2:27:28, 2:28:18 - 50s

Avg. 56s

O3-core-2.9

2:42:44, 2:43:21 - 37s

2:44:30, 2:45:09 - 39s

2:45:29, 2:46:09 -40s

Avg. 38s ~32% faster

Run on temurin-17.jdk. Used emr 3.6.0-SNAPSHOT for O3-core-2.8 and deployed 2.9.0-SNAPSHOT war for O3-core-2.9

Just noticed in logs that in 03-core-2.8 the attachements module didn’t start as it required 2.8.3 of core… Not sure what’s behind that. It might be even slower then on 2.8. In O3-core-2.9 it started correctly.

org.openmrs.module.ModuleException: Module requires version matching 2.8.3.  Current code version is 2.8.2
	at org.openmrs.module.ModuleUtil.checkRequiredVersion(ModuleUtil.java:396) ~[openmrs-api-2.8.2-SNAPSHOT.jar:?]

There’s apparently some issue with SDK performance. It wasn’t in scope of my work, but maybe worth investigating why Docker is much faster (at least on my machine by 20s!).

1 Like
16

Thanks @raff for this update.

I also got the attachments module startup failure, which was caused by the backend CI build not having run to deploy this commit.

Yes i also noticed that running in docker environments was twice as fast.