Improviding Speed of OpenMRS - During Development and support

I use the SDK on bare-metal not docker, however I am finding that starting and running servers at times becomes very slow especially when there is data loaded during support requests

My hardware is a Mac 2.4 GHz 8-Core Intel Core i9, 32GB RAM running Java 8 on Oracle JDK, I also have OpenJDK installed (if needed)

I am using MySQL 5.7 this InnoDB memory setting that uses up to 6GB

innodb_buffer_pool_size=6G

The Maven settings in my .bash_profile are

export MAVEN_OPTS="-Xms6144m -Xmx6G -noverify"

and I run my SDK as

mvn -T 4 openmrs-sdk:run -DserverId=ugandaemr

in an attempt to parallelize the builds and Tomcat running

The server spends about 10 minutes at

INFO - EhCacheManagerFactoryBean.destroy(181) |2022-01-25 09:24:24,745| Shutting down EhCache CacheManager
INFO - ServiceContext.startRefreshingContext(832) |2022-01-25 09:24:24,746| Refreshing Context
INFO - EhCacheManagerFactoryBean.afterPropertiesSet(129) |2022-01-25 09:24:40,330| Initializing EhCache CacheManager

I am never sure what is happening in here and we just have to wait, on the client side this takes anywhere from 3 to 4 hours

1 Like

With such a beast, and facing delays?? am marveled…!

Am imagining how long it will take a normal guy with normal tools :upside_down_face:

1 Like

I feel your pain @ssmusoke and would also love to see us all put our heads together to see if we can improve this. In my experience I’ve never found much difference in the size of the database - it is interesting if you find that your startup time is considerably slower when you are connecting to a DB with millions of records vs. a dev or test instance with far fewer. I’d probably take that as a separate issue to investigate on it’s own.

Spring application context startup and bean instantiation/wiring is generally the biggest culprit I believe, and I agree that it would be more helpful to have some information being spit into the logs that is showing this process at work rather than just sitting there at those INFO statements for many minutes (or hours!). It would be interesting if someone could identify some logging levels that could be enabled (eg. what classes/packages at what log levels) and would provide enough, but not too much, logging in this area to be able to gain some insights into what is taking so long. Starting up in a Profiler session is another approach that will likely gain insights, though that takes some experience to be able to interpret, and I’m no expert at it.

In the short term, what is “easy” and would be interesting and hopefully instructive to look at, would be for you to do an exercise like the following:

  1. Install a brand new SDK instance of ugandaemr, and move all modules out of the modules directory and into some temporary location (eg. modules-bak).

  2. Start up the server and note the time it takes to complete (eg. server listening on port 8080)

  3. Add modules back in one-by-one, in proper dependency order. Each time you add a module in, repeat step 2

Once you have done this with all of the modules, have a look at the results, and see if it identifies any particular modules that, when added, caused the spring startup time to dramatically increase. Report back here on the results - I’d find them very interesting to see, and we can continue brainstorming from there.

Thoughts? Mike

2 Likes

@mseaton those are the exact steps that i would take!

1 Like

btw @mseaton @ssmusoke @dkayiwa

Isn’t it high time we think of integrating OMRS with tracing tools like opentelemetry to track the time each process takes?

We could track database connection processes, module startups, request/response processes and many more, most especially now that openmrs is adopting a seemingly frontend-backend architecture.

With this tool working, finding the exact spot causing delay is somewhat simpler.

1 Like

:scream: :scream: :scream: Wow. Watching this thread with great interest. Thanks for raising this @ssmusoke.

@ssmusoke which version of the platform are you using? And can you provide the number (or a list) of modules.

@mseaton’s suggestions are on point. Our current approach on handling module startup is inefficient and doesn’t scale well. We are going to need to reinvent things a bit to support OpenMRS 3 deployment, which requires a discreet build step. I’m hoping we can leverage some ideas @raff has for foregoing our hot-swapping of modules in exchange for a backwards-compatible build process that could vastly improve startup times, reduce complexity, and improve the developer experience.

My openmrs-distro.properties file

# Sat Aug 13 11:04:47 EAT 2016
# SDK server properties file
# TODO: Remove the hard-coded versions and use property placeholders from the pom.xml file
omod.addresshierarchy=2.11.0
omod.metadatadeploy=1.11.0
omod.reportingrest=1.11.0
omod.serialization.xstream.type=omod
omod.serialization.xstream=0.2.14
omod.registrationapp=1.18.0
omod.event.groupId=org.openmrs
omod.event=2.9.0
omod.atlas.type=omod
omod.uilibrary=2.0.6
omod.htmlwidgets=1.10.0
omod.dataexchange=1.3.3
name=UgandaEMR
omod.appui=1.12.0
omod.allergyui=1.8.2
db.sql=classpath\://new-install.sql
omod.providermanagement=2.11.0
omod.formentryapp=1.4.2
omod.idgen=4.5.0
omod.metadatamapping=1.3.4
omod.htmlformentryui=1.10.0
db.h2.supported=false
omod.emrapi=1.29.0
omod.referenceapplication=2.10.0
omod.htmlformentry=4.3.0
omod.appframework=2.14.0
omod.referencemetadata=2.10.1
omod.coreapps=1.31.0
omod.reportingcompatibility=2.0.6
omod.reporting=1.23.0
omod.reportingui=1.6.0
omod.adminui=1.3.0
omod.registrationcore=1.9.0
omod.appointmentscheduling=1.12.0
omod.appointmentschedulingui=1.9.0
war.openmrs=2.3.1
omod.webservices.rest=2.28.0
omod.calculation=1.2
version=3.3.1-SNAPSHOT
omod.uicommons=2.12.0
omod.xforms=4.3.12
omod.uiframework=3.17.0
omod.metadatasharing=1.6.0
omod.dataintegrity=4.4.3
omod.databasebackup=1.3.0
omod.owa=1.10.0
omod.ugandaemr=3.3.1-SNAPSHOT
omod.ugandaemrreports=2.0.28
omod.ugandaemrfingerprint=1.0.11
omod.legacyui=1.8.2
omod.fhir2=1.2.2
omod.dataentrystatistics=1.7.0
omod.patientflags=3.0.2
omod.ugandaemrsync=1.1.2
omod.patientqueueing=1.2.0
omod.cohort=2.2.1
1 Like

Ouch that is a long list. Not a helpful comment but out of curiosity @ssmusoke, roughly what % of these modules do you think are basically unnecessary but are kicking around due to tricky dependencies (if any)? (Not meaning to be rude, I just remember you making a comment to this effect a while ago.)

Off the top of my head - these are the ones that could be merged and moved

  1. Appointmenscheduling and Appointment scheduling UI are required as part of Reference Application Module
  2. Xforms - added to fix some bug for forms not being displayed on the visit page
  3. Data Exchange - could be replaced by Iniz
  4. Form Entry App - I assumed its needed by HTML Form Entry never bothered to check again years later
  5. Allergy UI - could come in handy (not in use at the moment)
  6. OWA hanging dependency now removed

All other modules are required by reference application off which we build, and additional modules we need for our functionality

1 Like

Yeah, this is a continual problem… generally with our distributions startup time gets slower and slower while refreshing the context, and eventually we get sick of it and spend some time digging into it and speeding things up (which is never easy) and share a couple minutes off startup… and then, a few yesrs go by, and it slows down again.

What @mseaton suggests is spot-on.

Also, there’s the long-time issue where @Autowiring within services can really slow thing down for some reason we’ve never figured out. Any of the OpenMRS module should hopefully not be doing this, but worth checking out any of the custom modules. I tried to find the original link for this but couldn’t, but here’s one that talks about it:

Take care, Mark

1 Like

@jnsereko Sounds a good idea to me… It would help @devs track such issues of delays faster in abnormal stituations

1 Like

We have always noted very slow startup whenever we use annotations i.e. @Repository. To improve startup time, we have to replace most annotations and in the end have KenyaEMR start within 3-5min.

1 Like