Module Class Loader Issue

I’m using OpenMRS 2.0.6, and I’m having issues with linkage errors during class loading. Here’s the scenario:

  1.  Module A is deployed to OpenMRS with version 2..0.1 of javax.ws.rs-api
    
  2.  Module B is deployed to OpenMRS with version 2.1.0 of javax.ws.rs-api
    

Module A and B have no dependencies to each other. Module B only depends on the legacy-ui module.

I receive a Linkage Error sometimes when I try to start OpenMRS or run classes that have a dependency on the javax.ws.rs package.

I would rather not add a dependency from B to module A. Module B depends on multiple other modules, and I don’t want a user to have to load the other unneeded modules.

The class having the issues is the service implementation defined in the moduleApplicationContext.xml file. I performed some debugging, and the issues seems to happen when the Spring context is refreshed. I placed a breakpoint in the ModuleClassLoader loadClass method, and the refresh goes through there first. Everything looks correct at this point because the class is loaded from Module B’s class loader. Then a call comes from Spring refresh to the OpenmrsClassLoader loadClass method. It then looks up all module class loaders that have a reference to the javax.ws.rs package. It find 2 modules (Module A and B). But the list has Module A first, and tries to load the class and fails.

HTTP Status 500 - Handler processing failed; nested exception is java.lang.LinkageError: ClassCastException: attempting to castfile:/C:/windows/system32/config/systemprofile/Application%20Data/OpenMRS/.openmrs-lib-cache/ModuleA/javax/ws/rs/client/ClientBuilder.class to file:/C:/windows/system32/config/systemprofile/Application%20Data/OpenMRS/.openmrs-lib-cache/ModuleB/javax/ws/rs/client/ClientBuilder.class

Thanks, Steve

There could be something wrong with the way one of the modules is setup, a module’s dependencies are not visible to other modules unless you explicitly make one require another in the config.xml file.

Note that there could be transitive dependencies e.g module A has a library X, B has a library Z where libraries X and Z have a common dependency. I think you might want to use the dependency tree plugin to see whether the library is already transitively getting included in a duplicate way, plus you might to share you modules’ code for someone else to take a look.

Thanks Steve for reporting and debugging. I’m confused by you saying

and

which contradict each other.

Is it possible that B depends on C, which depends on A? If you are sure there are no require or aware of dependency between 2 modules, it is most likely a bug.

Could you please create an issue in JIRA? Can you reproduce the issue on each server startup or is it happening randomly? Are you in position to share your modules source code or create 2 simple basic modules to highlight the issue?

Sorry, I misstated the information. Module B only depends on the legacy-ui module. Module A depends on multiple other modules. There are also no aware of dependencies in either module, and there are no required dependencies between the two modules.

I’m currently creating two test modules to reproduce the issue. I’ll post the information once I’m able to create a reproducible test case.

Steve

1 Like

I do have a reproducible test case. Performing the following steps below will cause the linkage error to occur. Please advise if I have something in the modules configured incorrectly.

  1. Deploy OpenMRS Platform 2.0.6 (bundled with legacy-ui module).
  2. Open the Manage modules page from the OpenMRS admin page.
  3. Deploy Module A (https://github.com/sjmckee/Module-A.git).
  4. Deploy Module B (https://github.com/sjmckee/Module-B.git). a. Receive the linkage error during deployment of Module B: Constructor threw exception; nested exception is java.lang.LinkageError: ClassCastException: attempting to castfile:/C:/windows/system32/config/systemprofile/Application%20Data/OpenMRS/.openmrs-lib-cache/modulea/javax/ws/rs/client/ClientBuilder.class to file:/C:/windows/system32/config/systemprofile/Application%20Data/OpenMRS/.openmrs-lib-cache/moduleb/javax/ws/rs/client/ClientBuilder.class

Steve

I was able to reproduce the issue… The full stack trace is:

SEVERE: Servlet.service() for servlet [openmrs] in context with path [/openmrs] threw exception [Request processing failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceContext$child#23' defined in URL [jar:file:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/moduleb/moduleb.jar!/moduleApplicationContext.xml]: Cannot create inner bean 'org.springframework.transaction.interceptor.TransactionProxyFactoryBean#1f95ba86' of type [org.springframework.transaction.interceptor.TransactionProxyFactoryBean] while setting bean property 'moduleService' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.transaction.interceptor.TransactionProxyFactoryBean#1f95ba86' defined in URL [jar:file:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/moduleb/moduleb.jar!/moduleApplicationContext.xml]: Cannot create inner bean 'org.openmrs.module.moduleb.impl.ModuleBServiceImpl#6d31a95d' of type [org.openmrs.module.moduleb.impl.ModuleBServiceImpl] while setting bean property 'target'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.openmrs.module.moduleb.impl.ModuleBServiceImpl#6d31a95d' defined in URL [jar:file:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/moduleb/moduleb.jar!/moduleApplicationContext.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.openmrs.module.moduleb.impl.ModuleBServiceImpl]: Constructor threw exception; nested exception is java.lang.LinkageError: ClassCastException: attempting to castfile:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/modulea/javax/ws/rs/client/ClientBuilder.class to file:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/moduleb/javax/ws/rs/client/ClientBuilder.class] with root cause
java.lang.LinkageError: ClassCastException: attempting to castfile:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/modulea/javax/ws/rs/client/ClientBuilder.class to file:/C:/Users/Rafal/openmrs/mcl/.openmrs-lib-cache/moduleb/javax/ws/rs/client/ClientBuilder.class
        at javax.ws.rs.client.ClientBuilder.newBuilder(ClientBuilder.java:105)
        at javax.ws.rs.client.ClientBuilder.newClient(ClientBuilder.java:121)
        at org.openmrs.module.moduleb.impl.ModuleBServiceImpl.<init>(ModuleBServiceImpl.java:40)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1086)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1038)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1469)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1469)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
        at org.openmrs.module.ModuleUtil.refreshApplicationContext(ModuleUtil.java:842)
        at org.openmrs.module.web.WebModuleUtil.refreshWAC(WebModuleUtil.java:866)
        at org.openmrs.module.web.controller.ModuleListController.onSubmit(ModuleListController.java:196)
        at org.springframework.web.servlet.mvc.SimpleFormController.processFormSubmission(SimpleFormController.java:274)
        at org.springframework.web.servlet.mvc.AbstractFormController.handleRequestInternal(AbstractFormController.java:275)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:146)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.openmrs.module.web.filter.ForcePasswordChangeFilter.doFilter(ForcePasswordChangeFilter.java:60)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterException in thread "Thread-108" Exception in thread "Thread-115" Exception in thread "Thread-116" org.openmrs.module.ModuleException: Unable to run onStartup() method as Daemon
        at org.openmrs.api.context.Daemon.runStartupForService(Daemon.java:258)
        at org.openmrs.api.context.ServiceContext$1.run(ServiceContext.java:962)

I will continue to look into that.

@sjmckee do you mind if the versions of javax.ws.rs-api are the same?

The issue occurs even if they are the same version.

In order to be able to use javax.ws.rs.Client one would need to create it using its implementation instead of relying on ClientBuilder.newClient() to auto-discover it. The reason is that the newBuilder() called by newClient() does not use caller’s class loader, which is ModuleClassLoader, but the thread context class loader, which is always OpenmrsClassLoader.

Changing ClientBuilder.newClient() to JerseyClientBuilder.createClient(); does lead to yet another error, which I’m still investigating:

WARN - AbstractApplicationContext.refresh(487) |2018-01-31 16:26:21,045| Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceContext$child#0' defined in URL [jar:file:/C:/Users/Rafal/openmrs/mcl2/.openmrs-lib-cache/modulea/modulea.jar!/moduleApplicationContext.xml]: Cannot create inner bean 'org.springframework.transaction.interceptor.TransactionProxyFactoryBean#2af09425' of type [org.springframework.transaction.interceptor.TransactionProxyFactoryBean] while setting bean property 'moduleService' with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.transaction.interceptor.TransactionProxyFactoryBean#2af09425': Post-processing of FactoryBean's object failed; nested exception is java.lang.LinkageError: loader constraint violation: loader (instance of org/openmrs/util/OpenmrsClassLoader) previously initiated loading for a different type with name "javax/ws/rs/core/Response"
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1469)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:762)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
        at org.openmrs.module.ModuleUtil.refreshApplicationContext(ModuleUtil.java:842)
        at org.openmrs.module.web.WebModuleUtil.refreshWAC(WebModuleUtil.java:866)
        at org.openmrs.web.Listener.performWebStartOfModules(Listener.java:658)
        at org.openmrs.web.Listener.performWebStartOfModules(Listener.java:637)
        at org.openmrs.web.Listener.startOpenmrs(Listener.java:268)
        at org.openmrs.web.WebDaemon$1.run(WebDaemon.java:42)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.transaction.interceptor.TransactionProxyFactoryBean#2af09425': Post-processing of FactoryBean's object failed; nested exception is java.lang.LinkageError: loader constraint violation: loader (instance of org/openmrs/util/OpenmrsClassLoader) previously initiated loading for a different type with name "javax/ws/rs/core/Response"
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:133)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:302)
        ... 20 more
Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of org/openmrs/util/OpenmrsClassLoader) previously initiated loading for a different type with name "javax/ws/rs/core/Response"
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.springframework.util.ClassUtils.getMostSpecificMethod(ClassUtils.java:768)
        at org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource.computeTransactionAttribute(AbstractFallbackTransactionAttributeSource.java:144)
        at org.springframework.transaction.interceptor.AbstractFallbackTransactionAttributeSource.getTransactionAttribute(AbstractFallbackTransactionAttributeSource.java:100)
        at org.springframework.transaction.interceptor.TransactionAttributeSourcePointcut.matches(TransactionAttributeSourcePointcut.java:38)
        at org.springframework.aop.support.AopUtils.canApply(AopUtils.java:225)
        at org.springframework.aop.support.AopUtils.canApply(AopUtils.java:262)
        at org.springframework.aop.support.AopUtils.findAdvisorsThatCanApply(AopUtils.java:294)
        at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findAdvisorsThatCanApply(AbstractAdvisorAutoProxyCreator.java:118)
        at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findEligibleAdvisors(AbstractAdvisorAutoProxyCreator.java:88)
        at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.getAdvicesAndAdvisorsForBean(AbstractAdvisorAutoProxyCreator.java:69)
        at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.wrapIfNecessary(AbstractAutoProxyCreator.java:330)
        at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessAfterInitialization(AbstractAutoProxyCreator.java:293)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:422)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.postProcessObjectFromFactoryBean(AbstractAutowireCapableBeanFactory.java:1711)
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:130)
        ... 21 more

For the time being you can go around that particular error by not using javax.ws.rs classes in method declarations in beans. Basically read and process the response and return a result represented by your own class or any class from jdk or openmrs classpath.

I already tried doing that, but I ran into other issues. I commented out all my methods that returned a Response object and all my other methods returned my own Java objects, but it still failed. I think it complained about the use of the JsonMappingException object or something that’s just used within the methods. I’ve tried so many different iterations of changes to get around it, but haven’t been successful.

As we continue to investigate this, for a temporary ugly workaround, you can create a new module that just has the javax.ws.rs dependency. Then modules A and B depend on this module. Change the javax.ws.rs dependency for Modules A and B to scope provided.

That’s basically what I’m doing at the moment. I added the javax.ws.rs dependencies to Module A, removed the javax.ws.rs dependencies from Module B, and made Module B depend on module A. This seems to work fine, but it’s just annoying to create dependencies that shouldn’t be needed. Instead of just deploying just Module B, I now have to also deploy Module A and Module C because Module A depends on Module C (3 modules instead of just 1).

@dkayiwa Has this issue been worked out further? Because this issue still exists and crops up from time to time while trying to run OpenMRS

Not yet. Would you like to work on it?

@dkayiwa Yes, I would like to investigate the issue further and see where it is cropping up from.

Thanks and go ahead. https://issues.openmrs.org/browse/TRUNK-5344

1 Like