I use the SDK on bare-metal not docker, however I am finding that starting and running servers at times becomes very slow especially when there is data loaded during support requests
My hardware is a Mac 2.4 GHz 8-Core Intel Core i9, 32GB RAM running Java 8 on Oracle JDK, I also have OpenJDK installed (if needed)
I am using MySQL 5.7 this InnoDB memory setting that uses up to 6GB
innodb_buffer_pool_size=6G
The Maven settings in my .bash_profile are
export MAVEN_OPTS="-Xms6144m -Xmx6G -noverify"
and I run my SDK as
mvn -T 4 openmrs-sdk:run -DserverId=ugandaemr
in an attempt to parallelize the builds and Tomcat running
The server spends about 10 minutes at
INFO - EhCacheManagerFactoryBean.destroy(181) |2022-01-25 09:24:24,745| Shutting down EhCache CacheManager
INFO - ServiceContext.startRefreshingContext(832) |2022-01-25 09:24:24,746| Refreshing Context
INFO - EhCacheManagerFactoryBean.afterPropertiesSet(129) |2022-01-25 09:24:40,330| Initializing EhCache CacheManager
I am never sure what is happening in here and we just have to wait, on the client side this takes anywhere from 3 to 4 hours
I feel your pain @ssmusoke and would also love to see us all put our heads together to see if we can improve this. In my experience I’ve never found much difference in the size of the database - it is interesting if you find that your startup time is considerably slower when you are connecting to a DB with millions of records vs. a dev or test instance with far fewer. I’d probably take that as a separate issue to investigate on it’s own.
Spring application context startup and bean instantiation/wiring is generally the biggest culprit I believe, and I agree that it would be more helpful to have some information being spit into the logs that is showing this process at work rather than just sitting there at those INFO statements for many minutes (or hours!). It would be interesting if someone could identify some logging levels that could be enabled (eg. what classes/packages at what log levels) and would provide enough, but not too much, logging in this area to be able to gain some insights into what is taking so long. Starting up in a Profiler session is another approach that will likely gain insights, though that takes some experience to be able to interpret, and I’m no expert at it.
In the short term, what is “easy” and would be interesting and hopefully instructive to look at, would be for you to do an exercise like the following:
Install a brand new SDK instance of ugandaemr, and move all modules out of the modules directory and into some temporary location (eg. modules-bak).
Start up the server and note the time it takes to complete (eg. server listening on port 8080)
Add modules back in one-by-one, in proper dependency order. Each time you add a module in, repeat step 2
Once you have done this with all of the modules, have a look at the results, and see if it identifies any particular modules that, when added, caused the spring startup time to dramatically increase. Report back here on the results - I’d find them very interesting to see, and we can continue brainstorming from there.
Isn’t it high time we think of integrating OMRS with tracing tools like opentelemetry to track the time each process takes?
We could track database connection processes, module startups, request/response processes and many more, most especially now that openmrs is adopting a seemingly frontend-backend architecture.
With this tool working, finding the exact spot causing delay is somewhat simpler.
@ssmusoke which version of the platform are you using? And can you provide the number (or a list) of modules.
@mseaton’s suggestions are on point. Our current approach on handling module startup is inefficient and doesn’t scale well. We are going to need to reinvent things a bit to support OpenMRS 3 deployment, which requires a discreet build step. I’m hoping we can leverage some ideas @raff has for foregoing our hot-swapping of modules in exchange for a backwards-compatible build process that could vastly improve startup times, reduce complexity, and improve the developer experience.
Ouch that is a long list. Not a helpful comment but out of curiosity @ssmusoke, roughly what % of these modules do you think are basically unnecessary but are kicking around due to tricky dependencies (if any)?
(Not meaning to be rude, I just remember you making a comment to this effect a while ago.)
Yeah, this is a continual problem… generally with our distributions startup time gets slower and slower while refreshing the context, and eventually we get sick of it and spend some time digging into it and speeding things up (which is never easy) and share a couple minutes off startup… and then, a few yesrs go by, and it slows down again.
Also, there’s the long-time issue where @Autowiring within services can really slow thing down for some reason we’ve never figured out. Any of the OpenMRS module should hopefully not be doing this, but worth checking out any of the custom modules. I tried to find the original link for this but couldn’t, but here’s one that talks about it:
We have always noted very slow startup whenever we use annotations i.e. @Repository. To improve startup time, we have to replace most annotations and in the end have KenyaEMR start within 3-5min.
