GSoC 2021 - Export and Import feature for the OMRS reporting module: Updates

jayasanka · June 6, 2021, 3:52pm

@ibacher @dkayiwa Do you have any idea resolve this issue?

heshan · June 6, 2021, 4:56pm

Here’s a pastbin stack tree of the error I’m getting

k.joseph · June 7, 2021, 6:46am

Please share a full stack trace.

heshan · June 7, 2021, 6:53am

org.openmrs.api.context.ServiceContext.getService(ServiceContext.java:686)
org.openmrs.api.context.Context.getService(Context.java:986)
org.openmrs.module.dhisconnector.api.impl.DHISConnectorServiceImpl.exportMapping(DHISConnectorServiceImpl.java:1021)
org.openmrs.module.dhisconnector.api.impl.DHISConnectorServiceImpl.exportSelectedMappings(DHISConnectorServiceImpl.java:909)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
com.sun.proxy.$Proxy751.exportSelectedMappings(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:303)
org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.openmrs.aop.LoggingAdvice.invoke(LoggingAdvice.java:122)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
com.sun.proxy.$Proxy752.exportSelectedMappings(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
com.sun.proxy.$Proxy752.exportSelectedMappings(Unknown Source)
org.openmrs.module.dhisconnector.web.controller.DHISConnectorController.exportMapping(DHISConnectorController.java:208)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:177)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.module.web.filter.ForcePasswordChangeFilter.doFilter(ForcePasswordChangeFilter.java:60)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:73)
org.openmrs.module.owa.filter.OwaFilter.doFilter(OwaFilter.java:93)
org.openmrs.module.owa.filter.OwaFilter.doFilter(OwaFilter.java:61)
org.openmrs.module.web.filter.ModuleFilterChain.doFilter(ModuleFilterChain.java:71)
org.openmrs.module.web.filter.ModuleFilter.doFilter(ModuleFilter.java:57)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.web.filter.OpenmrsFilter.doFilterInternal(OpenmrsFilter.java:109)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:150)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.openmrs.web.filter.StartupFilter.doFilter(StartupFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:319)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)

k.joseph · June 7, 2021, 6:56am

please file the entire/whole stack trace and your code or PR to aid troubleshooting @heshan

heshan · June 7, 2021, 6:59am

Sure, give me 10 mins

heshan · June 7, 2021, 7:22am

Pull Request:

k.joseph · June 7, 2021, 8:02am

dialog moving to the attached PR. left you some feedback

heshan · June 7, 2021, 8:09am

Thank you, I replied, You can find the stack trace pastebin on the PR as well.

heshan · June 8, 2021, 7:48pm

2021-06-07T18:30:00Z Update

Today I continued experimenting with the current implementation amending it to find a way to export periodIdicatorReportDefinition alongside the DHIS mapping file.

I got it to export two additional XML files for periodIndicatorReport and DataDefinition. The functionality is working fine but I still have doubts on what would be the best way to export the files and in which format and what sort of a method to use to save them back in.

Looking forward to discussing these doubts at this weeks meeting. @k.joseph Let me know a time and a day if the currently scheduled one doesn’t work for you.

For now, Here’s is a small demo of what I’ve done so far.

heshan · June 9, 2021, 3:26pm

2021-06-08T18:30:00Z Update

The metadata sharing module is only able to export the metadata as serialized objects, if we follow that method we can get the serialized objects from the metadata module and export them as XML files. But in the same way, we could serialize the metadata object itself into XML and export those files. Should figure out what is the best way to do it.

Tomorrow I’ll be looking into that.

cc: @k.joseph

heshan · June 15, 2021, 6:41pm

3rd weekly meeting - 2021-06-13T18:30:00Z

Attendees:

@k.joseph @heshan

Notes

Heshan gave his update and demoed the period indicator report export and import methods
Kaweesi suggested few spots where the code could be improved
Heshan couldn’t get into saving objects because the deserialisation of xmls are failing
Is getting the serialised data directly from the metadata sharing module
Metadata module can be used to get serialised objects out but not period indicator reports or another type of objects
Kaweesi suggested to generate the same exact file as the metadata module instead of going for a different format
Compared the same object exported from this new implementation and metadata sharing module
- Meatdata module serialises all the objects into a one file by putting them as a list
Kaweesi pointed out that the creator attribute of the objects should be handled accordingly when importing the metadata into a system
Tentatively start by importing some metadata object exported from metadata sharing module from this end
Kaweesi suggested: For the entire report export just use one file to keep the report metadata

heshan · June 20, 2021, 4:16pm

2021-06-19T18:30:00Z Update

I studied the metadata sharing module today and tried to make a similar export in the DHIS module. The error I got when deserializing the XML files is still there. https://wiki.openmrs.org/pages/viewpage.action?pageId=3346315 I’m using the method in this thread.

Tomorrow I will be looking into that issue.

cc: @k.joseph @dkayiwa

heshan · June 21, 2021, 4:06pm

2021-06-20T18:30:00Z Updates

Tried to serialize and deserialize the serialized objects using jackson-dataformat-xml but it runs into a recursive object because creator attribute we have in SerializedObjects. Still trying to make a test program work properly as a MVP. Tomorrow I’ll dig deeper and find out what’s wrong with the previous way I tried.

heshan · June 23, 2021, 3:13pm

2021-06-22T18:30:00Z Update

I got my test functions to work properly today. Now it can export a mapping file with the corresponding period indicator report and import them back again.

Here’s a little progress demo video I made,

Tomorrow I’ll be working on adding the other underlying metadata like CohortDefinitions and DataDefinitions. cc: @k.joseph @dkayiwa

heshan · June 26, 2021, 4:41pm

4th weekly meeting 2021-06-23T18:30:00Z

Attendees: @k.joseph @heshan

Heshan demoed the finalised import and export feature
Ran into a 500 error on the import mapping bundle demo
Kaweesi went through the stack trace and gave some pointers to solve the problem
- An object we haven’t attached to any session is in the transient state. Since it was never persisted, it doesn’t have any representation in the database. Because no session is aware of it, it won’t be saved automatically.
- https://www.baeldung.com/hibernate-session-object-states

heshan · June 26, 2021, 4:45pm

2021-06-25T18:30:00Z Update

Still getting the transient object 500 error even after wrapping the code in a try-catch block and persisting the object inside the catch. Tomorrow I will be looking into the error and will try to find a solution for it.

heshan · June 28, 2021, 4:23pm

2021-06-27T18:30:00Z Update

I tried to use try-catch blocks and somehow persist the metadata objects, now it works but only when imported into a new instance. If I import it into the same instance I get this error because it’s trying to persist an object that is already existing (conflicting UUIDs). I think it solvable by not saving the object if the same UUID already exists. Tomorrow I’m gonna try that.

heshan · June 30, 2021, 4:43pm

2021-06-29T18:30:00Z Update

The export and import features are now working fine, it is able to export the mapping files and all the related metadata including period indicator report definitions, cohort indicator data set definitions, cohort definitions, dimensions, and indicators. It now can without any errors import those metadata too. the metadata show up in the relevant places in the reporting module UI. But for some reason, it doesn’t map the period indicator report with the other metadata and when I try to run the report I get this error.

org.openmrs.module.reporting.evaluation.EvaluationException: Failed to evaluate Error evaluating because:
Cannot find parameter 'endDate' in org.openmrs.module.reporting.dataset.definition.CohortIndicatorDataSetDefinition null

Off the top of my head, I’m thinking of adding a new header file into the export bundle and map the metadata properly on the import to solve this issue.

Tomorrow I’ll be trying to find out the root cause for this behaviour and try to solve the issue.

@k.joseph @dkayiwa any thoughts?

heshan · July 1, 2021, 6:15am