[solved] openmrs and mysql running high cpu with no activity

Hello all, We are having a problem on our openmrs + bahmni server, and I was hoping for help solving it. Today I noticed that bahmni is not working well (requests are not completing properly), and see that the server is cnostantly running at around 100% cpu, with java running at about 25% and mysqld the remaining 75%.

This problem persists even if I reboot the server and restart bahmni. I can’t find anything meaningful in the logs, but I tried some steps from here to get some info, collecting some thread traces. Here are some traces:

jstack.4719.161735.139610360.txt (58.5 KB) jstack.4719.161736.047383027.txt (58.1 KB) jstack.4719.161737.423928139.txt (51.2 KB) jstack.4719.161736.767589388.txt (47.5 KB) jstack.4719.161738.758607423.txt (49.3 KB) jstack.4719.161738.078519343.txt (52.4 KB) jstack.4719.161739.421755000.txt (58.5 KB) jstack.4719.161740.127359621.txt (61.2 KB) jstack.4719.161741.522899573.txt (58.3 KB) jstack.4719.161740.815040048.txt (58.5 KB)

I’ve also been running SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST; in mysql to try to see what queries may be running. Here aer some results: queries.txt (6.7 KB)

All I can notice is that all the stack traces seem to show a thread trying to close inactive visits: org.openmrs.module.emrapi.adt.AdtServiceImpl.closeInactiveVisits(AdtServiceImpl.java:128)

We are running bahmni 0.89, so bahmni modules are also running. I’ve tried stopping all of the bahmni services (reports, lab, event log service, atomfeed, pacs integration service, erp), but the problem persists. There are currently no other users on the system.

This is on a system that is in use. My next plan was to try closing all active visits for the day to see if this helps.

Thanks for any help, Matthew

I can confirm that this seems to have mostly fixed the problem - hopefully the issue was just something particular to one of the 64 visits I closed. Java and mysql are running at about 20% cpu on average. This is still a bit worrying, but hopefully it is okay. There are still 19 active inpatient visits.

mysqld is still making those same queries from time to time.

I’m not sure if there is anything I could look for in these visits on the DB?

edit: here is a new set of stack traces. Still looks a bit the same? Though the close inactive visit is just in 2 of the traces.

jstack.4719.174232.858246077.txt (44.4 KB)

jstack.4719.174234.297430425.txt (44.4 KB)

jstack.4719.174235.026052790.txt (53.4 KB)

jstack.4719.174233.568463547.txt (44.4 KB)

jstack.4719.174235.716659780.txt (44.4 KB)

jstack.4719.174236.376056287.txt (44.1 KB)

jstack.4719.174237.054028267.txt (56.1 KB)

jstack.4719.174237.743116965.txt (44.1 KB)

jstack.4719.174238.427549467.txt (44.1 KB)

jstack.4719.174239.144604274.txt (44.4 KB)

I just found the Scheduler in the Openmrs Administration, changing the Close Stale Visits Task from every 1 sec to every 1 hour solved the issue.

Moved to the #software:bahmni category.

@mdg583 I wanted to commend you for asking the question in a very clear way, with all the relevant supporting info. And further, thank you for marking the question as resolved, and giving the solution, which might help some future person who runs into the same issue.

Do you know where the “every 1 second” setting came from? Our demo server is set to close stale visits once per day (though I don’t know if that’s a demo-specific setting or one copied from real-world installs).

Hi Darius, thanks for your comments. I’m not sure how it happened, it must just have been a mistake. This is the kind of thing that could have been changed for testing while trying to enable auto closing of visits.