deployment issue

Software
1

I’m using ubuntu 22.04 on AWS and set up openjdk 8, mysql 8, tomcat 9, openmrs platform 2.4.5 I can successfully see the tomcat response for localhost:8080 but I keep receiving the 404 response when I attempt localhost:8080/openmrs. The dump of the recent catalina.out is found here: sudo tail -n 200 /var/log/tomcat9/catalina.out[2025-11-03 18:31:24] [info] Sta - Pastebin.com

2

The catalina.out file isn’t very helpful; it just says this:

[2025-11-04 16:26:37] [crit] One or more listeners failed to start. Full details will be found in the appropriate container log file
[2025-11-04 16:26:37] [crit] Context [/openmrs] startup failed due to previous errors

If you can upload the appropriate log file, it may be possible to give you better direction.

3

Does this help at all?

4

It looks like OpenMRS hasn’t started any initialization since the openmrs property files, etc., were not created. Note, the .OpenMRS folder does not get any files

5

There should be a file called something like localhost.<date>.log I think that has the relevant parts in it. The bit about the “one or more listeners failed to start” is concerning because there’s a listener that’s basically the “entry point” for the application, so if that’s failing, I wouldn’t expect anything downstream of that (like the application directory) to show up.

6

Thanks much…..here’s the dump of that file. cat localhost.2025-11-04.log 04-Nov-2025 16:26:36.763 INFO [main] org.apache.c - Pastebin.com

7

Well, that’s a new one!

So this is actually an error being thrown while trying to initialize the logging system. More specifically, it’s trying create a directory like .OpenMRS/configuration. It appears that the attempt to create the directory fails and so it throws an exception. The exception itself would be easy to code around, but not so much the failure to be able to write to a directory.

Normally, openmrs tries to create the application directory in ~/.OpenMRS, where ~ is the home directory of whatever user OpenMRS is running as, so whatever user Tomcat is started as. Most likely, the failure to create a directory is down to a couple of possibilities:

  1. The user does not have a configured home directory. You might be able to get the tomcat user’s home directory (if you don’t know it and presuming you have a user named tomcat with something like getent passwd | grep tomcat | cut -d: -f6
  2. The user does have a home directory configured, but cannot actually write to it. This is often the case with Linux daemon accounts that are just meant to run services. In that case, OpenMRS will not be able to create a directory where it expects and you will have to use one of the methods described here to set the OpenMRS application directory to somewhere else.

2 is way more likely than 1 and of the three options in the page I linked, it’s usually better to change the CATALINA_OPTS since that’s the least invasive.

So something like:

mkdir -p /var/lib/openmrs # substitute whatever path suits your purpose
chown tomcat:tomcat /var/lib/openmrs # subsitute with the user name of the user Tomcat runs as

I’m not exactly sure about Tomcat 9, but in previous versions of Tomcat, there’s a script called setenv.sh that’s meant for this kind of thing. You’d just add a line to it like this:

CATALINA_OPTS="$CATALINA_OPTS -DOPENMRS_APPLICATION_DATA_DIRECTORY=/var/lib/openmrs"

With that setup, all your files will be in /var/lib/openmrs (which is actually where they’re supposed to be when the home directory isn’t writable).

8

I created the setenv.sh file as you mentioned and restarted. Still the files that should be in openmrs are not there. I put the setenv.sh file in /usr/share/tomcat9/bin

9

Is that in the same directory as catalina.sh? Also, is there any output to the localhost.04-Nov-2025.log file? I easily could’ve misdiagnosed things…

10

setenv.sh is in the same directory as catalina.sh (/usr/share/tomcat9/bin). Here’s the latest tail for localhost.2025-11-05.log: pastebin

11

Looks like the same issue. Quick q: you said you “created” the setenv.sh file as in it wasn’t there? Do you know that the $CATALINA_BASE environment variable is set to?

If not, and assuming Tomcat is a standard systemd unit service, it’s probably in the output of cat $(systemctl show -P FragmentPath tomcat.service).

The file needs to be at $CATALINA_BASE/setenv.sh to be picked up.

12

Tomcat indicated CATALINA_BASE: /var/lib/tomcat9…I noted the other tomcat .sh files are at /usr/share/tomcat9/bin so I placed setenv.sh there, as well. I copied setenv to the folder /var/lib/tomcat9 and still get the same issue mentioned in the localhost log file. My only constraint is to use refapp 2.x Would you have a tech stack you use that might easily work on aws?

Many thanks for your assistance!

13

Apologies in the delay here. It really seems like it should all work.

For the setenv.sh. Sorry I wasn’t specific enough. It should go in $CATALINA_BASE/bin. Although, I think you’re location might’ve worked as well (script is here, but it looks like it tries to read $CATALINA_BASE/bin/setenv.sh and if that doesn’t exist, tries to load $CATALINA_HOME/bin/setenv.sh; $CATALINA_HOME should be the folder that contains the /bin/catalina.sh script, but for w/e reason they prefer $CATALINA_BASE if that’s set to something else; it defaults to $CATALINA_HOME).

I don’t have anything I know works on AWS well off-hand and this is mostly down to my being somewhat unfamiliar with how it configures Tomcat… You’ve got the right tech stack I just don’t understand why it isn’t working.

14

Alright! So @levine and I met to review what was happening. It seems to be related to how Ubuntu (or other Debian-based distributions) configure their default install of Tomcat 9 (and future versions). Specifically, the changes added in this commit.

With that change in place, Tomcat is configured to only write to certain fixed directories: /var/lib/tomcat9/conf/Catalina, /var/lib/tomcat9/logs, /var/lib/tomcat9/webapps, /var/lib/tomcat9/work. These are the directories that Tomcat itself needs to write to, but applications like OpenMRS often need to write to other directories (though we try to keep things confined to just the OpenMRS Application Directory).

To address this, we needed to do a few steps:

  1. Create a directory for the OpenMRS Application Directory.
sudo mkdir -p /var/lib/openmrs
sudo chown tomcat:tomcat /var/lib/openmrs  # tomcat is the default user on Ubuntu
  1. Create the directory /bin in $CATALINA_BASE. This is where Tomcat tries to load setenv.sh from. On Ubuntu, $CATALINA_BASE is /var/lib/tomcat9, so this looks like:
sudo mkdir -p /var/lib/tomcat9/bin
  1. Inside the newly created directory, we create a file called setenv.sh:
sudo touch /var/lib/tomcat9/bin/setenv.sh
sudo chmod +x /var/lib/tomcat9/bin/setenv.sh
  1. Open that file in your favorite editor and set it’s contents to:
CATALINA_OPTS="$CATALINA_OPTS -DOPENMRS_APPLICATION_DATA_DIRECTORY=/var/lib/openmrs"
  1. Create a service override definition file for the Tomcat 9 service:
sudo mkdir -p /etc/systemd/system/tomcat9.service.d/
sudo touch /etc/systemd/system/tomcat9.service.d/override.conf
  1. Open that file in your favorite editor and set its contents to:
[Service]
ReadWritePaths=/var/lib/openmrs
  1. Update systemd to be aware of your changes and restart Tomcat:
sudo systemctl daemon-reload
sudo systemctl restart tomcat9

That seems like a lot of steps. Basically what we’re doing is creating a directory to serve as the OpenMRS Application Directory, which OpenMRS needs to write to. We then tell OpenMRS to use that directory as the application directory. Finally, we override the default configuration to tell systemd that Tomcat should be allowed to write to that directory.

15

Oh yikes!!! Too much work of something as simple as creating a directory to serve as the OpenMRS Application Directory, which OpenMRS needs to write to. :melting_face:

Thank you so much @ibacher for sharing. :+1:

16

I was originally going to suggest that maybe we edit the algorithm here so that it could fallback to one of the directories Tomcat can write to, but I don’t think it’s feasible to do so (none of the directories it can write to are really intended to store long-term application-specific data like we need). The fix proposed here (create a $HOME directory for the Tomcat user and modify the default service configuration to allow Tomcat to write files to it) would resolve the issue.

It turns out there’s an easier way to do steps 5 & 6 in the above:

  1. Run sudo systemctl edit tomcat9.service
  2. Set the contents to the below and save the file:
[Service]
ReadWritePaths=/var/lib/openmrs