I know it doesn’t help you know now, but I few tricks that have worked before to diagnose this kind of behaviour:
- Take thread dumps of the java process. If in linux or mac, send a
kill -3 <pid>, let’s say every 30 seconds. The thread dumps will appear on the tomcat logs, the one for stdout. A tool like https://java.net/projects/tda can help investigate what the threads were actually doing or waiting for, as well as deadlocks. It’s a non-blocker process, so it’s quite alright to run it anytime, even if there’s a chance of recovering. - It’s nice to configure the the JVM to print information about its garbage collection. A set of useful options:
-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintConcurrentLocks -verbose:gc. Every invocation to the garbage collector will print a line on the logs, and you can see if it’s doing a lot of full GCs and not freeing any memory (I’m used to call it ‘GC death’ or ‘GC catalepsy’).