Application Stuck at WARN - IdgenTask.run(32)

I’m using OpenMRS Platform 1.11.4, ReferenceApplication 2.3.1 and bundled modules.

I’m testing load balancing and session replication for the OpenMRS application and the setup is as follows:

  • Two tomcat containers holding deploying openmrs.war
  • Tomcat containers sharing the data directory containing modules(.omod files) and the openmrs-runtime.properties files.

I fail to run both the containers at the same time for long and on checking the logs found this: WARN - IdgenTask.run(32) |2017-07-07 03:03:09,338| Not running scheduled task. DaemonToken = null; enabled = false

1 Like

@themoonraker13 have you checket the tomcat config file to set the port numbers.Am not sure about that error but usually if you are running multiple servers then they should be requested at different port numbers e.g 8080,8081 and so on… Otherwise you may run into issues

Yes of course. I’ve a custom server.xml and tomcat containers are accessible on ports 8080 and 8081 respectively.

You see when I perform initial setup of openmrs, I do it for one tomcat container only and later restart both the containers to take up shared modules and database.

Also doing the above works well as I’m able to access one of the two containers at a given instance while other is stuck at WARN - IdgenTask.eun(32).

When I kill the tomcat that’s accessible, the tomcat stuck at WARN - IdgenTask.eun(32), all of a sudden resumes deployment and is successfully deployed and accessible.

Which version of tomcat are you running?

Apache Tomcat 8.0.41

@themoonraker13, as far as i can tell, there is nothing in the idgen scheduled task that should cause you issues here. My guess is that this is simply the last thing logged, but the actual issue lies elsewhere (this warning is generally expected if the scheduled task runs before the context is fully refreshed I believe).

Since you are running 2 instances of OpenMRS with a single database, it seems likely that there is some sort of thread contention, where a lock is likely getting held and is unable to be released. One area to investigate would be liquibase changesets. Can you step through startup with a debugger, or increase the log level to Debug for the entire application, and see if there is any more information?

Mike

I was sensing something related Database Contention. :thinking:

And, FYI, I’m not aware of anyone running OpenMRS with load balancing across tomcat servers. So you’re forging a new path here.

I vaguely recall @wyclif did some analysis of this in the past, and found that there are some specific things happening in OpenMRS that won’t work for load balancing, and someone will need to analyze and fix. @wyclif, is that right?

createAndRunTask is synchronized, if both Tomcats share the same JVM I guess one thread is running the method and prevents the other instance of IdgenTask to execute. Maybe it’s a feature, not a bug :slight_smile:

@darius, I’ve a little complex setup here. You see I’m using docker containers as follows:

  1. One MySQL docker container to contain openmrs database
  2. Two Tomcat docker containers with OpenMRS deployed in it accessing the single opnemrs database available in MySQL container
  3. One Apache Server docker container to handle load balancing and session replication

@lluismf, the tomcat docker containers have their own Java and have the application is deployed at a unique jvmRoute (set in server.xml)

Yes I did but with more emphasis on clustering

@darius and @lluismf, I was able to fix the issue by using two database users - separate user for each openmrs instance.