EMR API and Registration Core builds are hung

Development
1

I was on ci.openmrs.org today and noticed that both the Registration Core and EMR API jobs were hung… I stopped them and ran them again, but they still appear to be hung. I’ll include a snippet of the error log below. @aman @dkayiwa I saw there were some recent changes for the PostgreSQL project, I’m wondering if they are related? The errors appear to be with loading test data in the H2 database:

Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" java.lang.ExceptionInInitializerError
	at com.ibm.icu.text.Collator.<clinit>(Collator.java:946)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.h2.value.CompareMode.<clinit>(CompareMode.java:57)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:84)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Caused by: java.lang.IllegalArgumentException: Invalid version number: Version number may be negative or greater than 255
	at com.ibm.icu.util.VersionInfo.getInstance(VersionInfo.java:191)
	at com.ibm.icu.impl.ICUDebug.getInstanceLenient(ICUDebug.java:65)
	at com.ibm.icu.impl.ICUDebug.<clinit>(ICUDebug.java:69)
	... 15 more
Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" java.lang.NoClassDefFoundError: Could not initialize class org.h2.value.CompareMode
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:84)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Exception in thread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" java.lang.NoClassDefFoundError: Could not initialize class org.h2.value.CompareMode
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:84)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Exception in thread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@61364b5f" java.lang.NoClassDefFoundError: Could not initialize class org.h2.value.CompareMode
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:84)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:92)
	at org.h2.Driver.connect(Driver.java:72)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPerTaskAsynchronousRunner$TaskThread.run(ThreadPerTaskAsynchronousRunner.java:255)
2

Tangentially, I’m wondering if there’s way to tell Bamboo to automatically fail if a job seems hung after some number of minutes…

1 Like
3

The two compile fine, locally. Could be a bamboo problem.

1 Like
4

@dkayiwa @mogoodrich , Just gave a quick search and I think the problem may be due to this https://jira.atlassian.com/browse/BAM-21018 .

1 Like
5

Thanks @aman, that appears to be it!

Looks like at least some version of H2 is incompatible with the latest release of OpenJDK (262, released on July 14th):

https://wiki.openjdk.java.net/display/jdk8u/Archive

I’m running OpenJDK 1.8.0_265 and it’s failing for me locally as well.

Not sure what the best next steps are… it may be that those modules are building against old versions of OpenMRS Core, which include an old version of H2… I’m trying to narrow it down now… this will be a bit of a blocker though…

fyi @ibacher @mseaton @burke

Take care, Mark

6

I’ve seen this error locally and been trying to ignore it. H2 uses the icu4j library which uses the Java version to determine what version of Unicode it should support. Unfortunately, they don’t support version parts over 255. As Mark indicated, this assumption fails for recent versions of Java 8.

The issue has been reported to the ICU4J team as ICU-21219 and separately as ICU-21208, but those issues seem to be unassigned.

Conceptually, the easiest fix is to downgrade Java on those machines, but I assume that we’re simply using the package manager’s latest version and version-pinning adds its own headaches.

7

Thanks @ibacher… what is interesting is that I haven’t been able to narrow it down… I was thinking that perhaps this issue only manifest itself in older versions of the H2 library, but I ran HTML Form Entry (which builds against old versions of OpenMRS) and it passes.

Rolling back to OpenJDK 8u252 seems like the way to go in the short term, but we’d need a long-term fix as well.

I filed a ITSM ticket here: https://issues.openmrs.org/browse/ITSM-4280

fyi @cintiadr

Take care, Mark

8

Okay… I found it. We’re depending on an 11 year old version of ICU4J indirectly in the Event module by depending on a 9 year old version of ActiveMQ (the bug was fixed in ICU4J in 2010). There’s a major breaking change in ActiveMQ’s packing starting with 5.8.0, but it’s seems pretty straight-forward to upgrade to ActiveMQ 5.7.0 where all unit tests pass and the dependency on ICU4J is dropped. H2 seems to have a fallback mode for when ICU4J isn’t available on the classpath, so I think this should be fine.

Basically, we need to bump the ActiveMQ version in the events module, cut a new release, and then update whatever components depend on the events module to depend on the 2.8.0 release.

Objections? Comments?

Thanks!

3 Likes
9

@ian you are my hero for tracking that down! :slight_smile:

I’d say go for it… let me know if you need anything from me!

10

Oh that’s unusual! I don’t know how easy it will be to keep older JDK (not sure if it’s available on my installer) BUT I can stop recreating all bamboo agents and we can select previously installed versions of the JDK.

Can you give me the builds? Should be as simple as adding a proper requirement to the actual JDK update and setting the JAVA_HOME.

11

No worries, @cintiadr… @ibacher was able get a fix in, I believe… I think he closed the ITSM ticket as well… thanks!