I have a scheduled task in the xdssender module that runs every hour, for some reason after a certain amount of time, (like 1 day) the task stops, and you can’t restart it again in the UI. The only way I have figured to get the task running again is to restart openmrs completely.
(1) is there a way to restart a scheduled task without having to utiliize the interface ? (2) is there a way to monitor and automatically restart services that have gone down?
Thanks,
Christina
Is it possible to share the server side log and we try to figure out why this task stops and can’t be restarted?
Daniel,
Its been difficult to find where in the log there is an error because it shuts down randomly. Is there a way i retrieve logs from a previous day? I can track back to the last successful run time, but i can’t find the stored logs. Where would i locate them?
Are you able to just zip the entire logs folder and send it to me?
here is the error i can ascertain from the logs, its the last error for this process and then nothing.
ERROR - PullNotificationsTask.execute(31) |2022-06-01 15:49:11,266|
java.lang.ArrayIndexOutOfBoundsException
org.openmrs.api.APIAuthenticationException: Privileges required: Manage Global Properties
at org.openmrs.aop.AuthorizationAdvice.throwUnauthorized(AuthorizationAdvice.java:116)
at org.openmrs.aop.AuthorizationAdvice.before(AuthorizationAdvice.java:97)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy185.saveGlobalProperty(Unknown Source)
at org.openmrs.module.santedb.mpiclient.configuration.MpiClientConfiguration.getOrCreateGlobalProperty(MpiClientConfiguration.java:120)
at org.openmrs.module.santedb.mpiclient.configuration.MpiClientConfiguration.getProxy(MpiClientConfiguration.java:164)
at org.openmrs.module.santedb.mpiclient.api.impl.FhirMpiClientServiceImpl.getClient(FhirMpiClientServiceImpl.java:117)
at org.openmrs.module.santedb.mpiclient.api.impl.FhirMpiClientServiceImpl.exportPatient(FhirMpiClientServiceImpl.java:515)
at org.openmrs.module.santedb.mpiclient.api.impl.MpiClientServiceImpl.exportPatient(MpiClientServiceImpl.java:200)
at sun.reflect.GeneratedMethodAccessor942.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy309.exportPatient(Unknown Source)
at sun.reflect.GeneratedMethodAccessor942.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.openmrs.aop.LoggingAdvice.invoke(LoggingAdvice.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy310.exportPatient(Unknown Source)
at sun.reflect.GeneratedMethodAccessor942.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy310.exportPatient(Unknown Source)
at org.openmrs.module.santedb.mpiclient.aop.PatientUpdateWorker.run(PatientUpdateWorker.java:73)
ERROR - FhirMpiClientServiceImpl.exportPatient(530) |2022-06-01 15:49:11,293| org.openmrs.api.AP
could this be happening if the scheduled task is trying to run before the previously executed one finishes?
and is there a way to restart this task without having to restart the server entirely?
Is the zip of the logs folder still too big?
Here are the logs for today. The last successful run was 02 June 2022 04:40:05 EDT
isantepluslog.txt (3.32 MB)
And what is the name of the class for this task?
Pull point notifications
I mean the actual class name including the package.
There isn’t much to go on in the logs, but it is being spammed with these messages:
ERROR - PullNotificationsTask.execute(31) |2022-06-02 04:45:02,755|
java.lang.ArrayIndexOutOfBoundsException
There seems to be some step that might queue new messages, but there doesn’t seem to be a process for clearing an invalid message that gets added to the queue. So at a guess, what happens is that at some point the system receives a message it can’t process (due to the ArrayIndexOutOfBoundsException), so processing fails and the message gets logged. Then when the process starts again, it asks for new messages, gets the failing message, fails on it again and logs the message again. Probably restarting OpenMRS also flushes the queued message so things appear to work until another invalid message gets generated… But there’s not a lot to go on.
I’m kind of surprised that there’s not more detail in the log, since I would have thought that the call LOGGER.error(e.getMessage(), e) would at least log the stack trace and not just the method.
Notifications are pulled from from the Pull Point , parsed into HL7 messages and added to a List for processing.
So Given that this is had to reproduce ,and the Logger doesnt give much details, its had to tell where the error exactly occurs . Either at Parsing or Processing .
Can we just clear the entire List on throwing an error and Attept to fetch new messages from the Pull Point ??
Because on restarting the instance. Things work fine untill the next failure. I havent got a chance to see this failure and look at the logs my self
What was the name of the log file from which the above stack trace was got?
@dkayiwa , The Logs from the log file are not very specific to this issue. The other problem is that the issue is not very direct to reproduce.
The reason for asking the log file name is simply because different log files have different levels of detail.