I have just been troubleshooting an installation that has run out of data on a drive to find that the openmrs-lib-cache folders are created and are currently 7GB over less than 2 months.
I am wondering why these are created continuously in Tomcat - and whether there is a setting to avoid creating these temporary work folder.
@ssmusoke, these folders are created and used by the OpenMRS classloader, so I don’t believe there is an option to avoid creating them. You can look through the OpenmrsClassloader java class and see how and when these folders are created, and also what code exists to try to clean them up.
I know there were previous issues reported and fixed as you describe them. In particular, have a look at https://issues.openmrs.org/browse/TRUNK-1976
I’m assuming that you are running a version of core that has this fix applied to it, but it might be worth starting your investigation with this in mind.
Mike
Thanks @mseaton We are running 1.11.6 on the current server so I will watch it for a few days to see what happens then I will report back with our finding.
@ssmusoke i have made a commit to the 1.11.x branch. Feel free to test it out again and report if you get any more problems. Just to give some background, every time openmrs runs, it creates a new random temporary folder for the lib cache. The code tried to delete the previous lib cache but made a wrong assumption that the current temporary folder is the same as for the previous run. The commit i have made uses a fixed folder named “.openmrs-lib-cache” in the application data directory. That way, a running instance will always know where the cache was for the last runs and hence delete as expected. I used the application data directory because it was the easiest way i could think of to ensure that different running openmrs instances do not delete each other’s lib cache folder when using a fixed instead of random lib cache folder, since each one has a separate application data directory.
I think we made a mistake with the openmrs log file too by placing it in the application data dir but I personally don’t think that logs and the lib cache should be in the application data directory since they are technically not data. How about keeping the lib cache folder in the temp directory but with a predetermined name?
@wyclif that will be great, as long as we solve the problem of different openmrs instances overwriting each other’s lib cache. Do you have an off the head tip for addressing that?
I would say do the same as we do for the application data directory, i.e maintain a different directory name for each instance of the webapp based on the web app name
That would still result into problems when one runs the standalone version of openmrs with the same web app name as the non standalone version. Yet for the application data directory strategy, the standalone’s app data directory would be different from that of the non standalone instance.
Excuse my ignorance but is this cache of libraries really needed? The operating system already has a cache, so if these JARs are frequently read there’s a high probability that its pages are in RAM. Is there any test that proves that using this technique the app runs much faster?
I suggest to load the classes (or any other resource) from the JAR directly instead of expanding it and creating all those folders. Loaded classes are already cached in memory so it’s guaranteed not to read the same class more than once.
The servlet container locks all jar files loaded by URL classloaders, so we need to avoid this for module restart to work by creating a cache location than can be updated without getting locked by the servlet container. But I would like to backtrack on this based on this it seems the proper way to avoid jar file locking is to set antiJARLocking to true.
What about deleting files on system shutdown?
-Darius (by phone)
@darius the code that deletes files on system shutdown has always been in place for years. But the challenge is that this works only when the system is properly shutdown, which is not always the case, resulting into the reported problem on this thread.
If the JARs are locked they can’t be deleted or overridden, which is fine in a PROD environment (stopping the app server to replace it is enough). If the classloader needs just to use them read-only (which I believe is the case) it shouldn’t be a problem. Getting a URL of a class inside a JAR is a one-liner.
@dkayiwa I would like to agree with @wluyima and also suggest that the lib-cache folders & openmrs log files be moved out of the Application Data directory.
The log files tend to grow especially when the systems are used for a long time, and I am “SCARED” of asking users to delete those files in case of issues.
Thoughts?
@ssmusoke when in the application data directory, they are deleted every time openmrs shuts down or starts up because they are in a fixed known location. So they will never grow.
@dkayiwa in production i doubt that OpenMRS is shutdwn regularly, besides log and cache files are not data to reside in the app data directory.
