Error message on attempt to update to version 0.86

Currently I am playing around with Bahmni as we consider using it for a charitable hospital in the Punjab. While I was investigating the patient merge feature of OpenMRS I had to try out to update Bahmni. I ran into problems and report it in case one of your real clients run into this issue as well.

Observed result

Ansible task “Remove maintenace enable page from /var/www/html” fails because it cannot access port 80 during update to Bahmni version 0.86. This could be reproduced for the update from 0.83 and 0.85.

http://pastebin.com/D4FMVaNA

Expected result

The update to version 0.85 worked fine. Here is how it should look like: http://pastebin.com/fR06920N

Steps to reproduce

{
# Workaround mentioned in https://bahmni.atlassian.net/wiki/display/BAH/Install+Bahmni+on+CentOS
export postgres_repo_rpm_name="pgdg-centos92-9.2-7.noarch.rpm"

# Setup instructions from https://talk.openmrs.org/t/how-to-update-bahmni-to-latest-version/7731
yum remove bahmni-installer -y
yum install https://dl.bintray.com/bahmni/rpm/rpms/bahmni-installer-0.86-66.noarch.rpm -y
bahmni -i local install
}

Environment

  • CentOS 6.8 on VirtualBox (without Vagrant)
  • Host sytem: Ubuntu 16.04

Diagnostic information

Problem analysis

The problem is that the Ansible jobs fail to release port 80 after stopping the httpd service. I have found this out by adding pauses into the following scripts and then I have logged into the VM with another terminal. The funny thing is that the httpd service is stopped but still port 80 is occupied.

[root@kch ~]# service httpd status
httpd is stopped
[root@kch ~]# nmap -sT -O localhost

Starting Nmap 5.51 ( http://nmap.org ) at 2016-11-28 03:09 IST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000061s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 994 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http

If I kill the process manually within the pauses indicated below with the following command, the installation will succeed.

kill $(lsof -t -i:80)

janux@ultrabook-jm:bahmni-playbooks$ git diff
diff --git a/roles/httpd-maintenance/tasks/main.yml b/roles/httpd-maintenance/tasks/main.yml
index 7993a33..2ecc704 100644
--- a/roles/httpd-maintenance/tasks/main.yml
+++ b/roles/httpd-maintenance/tasks/main.yml
@@ -20,8 +20,10 @@
    state=absent
   when: maintenance_status == 'end'
 
+- pause: minutes=10
+
 - name: reload_httpd_service
   service: name=httpd state=restarted
   delegate_to: "{{ item }}"
   with_items: "{{groups['bahmni-emr']}}"
-  when:  maintenance_status == 'end'
\ No newline at end of file
+  when:  maintenance_status == 'end'
diff --git a/roles/httpd/tasks/main.yml b/roles/httpd/tasks/main.yml
index f560b22..a7ac82a 100644
--- a/roles/httpd/tasks/main.yml
+++ b/roles/httpd/tasks/main.yml
@@ -58,6 +58,8 @@
       - run_bahmni_web
       - stop_bahmni
 
+- pause: minutes=10
+
 - name: Start httpd
   service: name=httpd state=started enabled=yes
   tags:

Misc system info

  • SELinux is disabled.
1 Like

Hi @janux,

This is awesome analysis. We faced this problem few times while testing fresh install of Bahmni 0.86. But, we couldn’t reproduce it. Which version of Ansible is installed on your system?

Hi @ajjaswanth, I am kind of doing such things all day long. :slight_smile: :cry: :smiley: In case you are interested, here is the process I am trying to follow.

My Ansible version is 2.2.0.0. What might have caused the problem is that I am running an CentOS 6.8 while your pipeline seems to be running on CentOS 6.7. The problem with 6.8 is that it is not available for download on the official website. I think DigitalOcean also offers only 6.8/7.X.

1 Like

The Ansible version at the start (my 0.83 snapshot) is 2.1.1.0.

The problem is not caused by the CentOS or Ansible version.

Experiments

Start of update with Ansible 2.2.0.0

Experiment

(1) Load snapshot 0.83 (2)

{
# Workaround mentioned in https://bahmni.atlassian.net/wiki/display/BAH/Install+Bahmni+on+CentOS
export postgres_repo_rpm_name="pgdg-centos92-9.2-7.noarch.rpm"

# Setup instructions from https://talk.openmrs.org/t/how-to-update-bahmni-to-latest-version/7731
yum remove bahmni-installer -y
yum install https://dl.bintray.com/bahmni/rpm/rpms/bahmni-installer-0.86-66.noarch.rpm -y
bahmni -i local -av 2.2.0.0 install
}

Result

Resulted in the same problem.

TASK [httpd-maintenance : reload_httpd_service] ********************************
task path: /opt/bahmni-installer/bahmni-playbooks/roles/httpd-maintenance/tasks/main.yml:23
failed: [localhost -> localhost] (item=localhost) => {"failed": true, "item": "localhost", "msg": "[Sun Dec 04 21:37:58 2016] [warn] module cache_module is already loaded, skipping\n[Sun Dec 04 21:37:58 2016] [warn] module disk_cache_module is already loaded, skipping\n[Sun Dec 04 21:37:58 2016] [warn] module wsgi_module is already loaded, skipping\nhttpd: apr_sockaddr_info_get() failed for kch\nhttpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName\n(98)Address already in use: make_sock: could not bind to address [::]:80\n(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80\nno listening sockets available, shutting down\nUnable to open logs\n"}
        to retry, use: --limit @/opt/bahmni-installer/bahmni-playbooks/all.retry

Unset `KeepAlive` in the Apache configuration

Experiment

`vi /etc/httpd/conf/httpd.conf`

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

Result

Same problem

Run the update on CentOS 6.7

Experiment

  1. Download 6.7 Image

    http://mirror.nsc.liu.se/centos-store/6.7/isos/x86_64/CentOS-6.7-x86_64-minimal.iso

  2. Configure system

    • disable selinux
    • activate eth0
    • allow root login
  3. Install Bahmni 0.85

    • Timezone: Europe/Berlin

yum install https://dl.bintray.com/bahmni/rpm/rpms/bahmni-installer-0.85-91.noarch.rpm bahmni --help curl -L http://tiny.cc/defaultsetup >> /etc/bahmni-installer/setup.yml # Set timezone Europe/Berlin vi /etc/bahmni-installer/setup.yml bahmni -i local install

  1. Update to Bahmni 0.86

    {
    export postgres_repo_rpm_name="pgdg-centos92-9.2-7.noarch.rpm";
    yum remove bahmni-installer -y;
    yum install https://dl.bintray.com/bahmni/rpm/rpms/bahmni-installer-0.86-66.noarch.rpm -y;
    bahmni -i local install;
    }
    

Results

Same problem.

Hi @janux,

We are facing this issue when doing fresh two machine install (active/passive, master/slave). But, we are not facing this issue in fresh single machine install. You check the pipelines here https://ci.mybahmni.org. Login with GitHub.

One solution that we are thinking of is to replace the httpd restart with httpd reload. That should fix it for now. We are still trying to find out the root cause.

@janux, We changed from restart to reload. https://github.com/Bahmni/bahmni-playbooks/commit/e6fe6ee7013e31c1bf7ac3604c4b127aaf4a8899. Our deployment tests are running fine now.

I too was trying to upgrade to 0.86 and have CentOS 6.8 and getting this error -

Should I downgrade the OS to 6.7?

Thanks, Ashok

@ashokraman, Did you run the bahmni command as root?

Not sure if I did sudo su, but it does ask for the sudo password upfront - tried several times including the reload, restart change listed above in any case, thanks,

Your error looks like because of not running sudo su and when it was trying to install bahmni-emr rpm. The actual httpd error (TASK [httpd-maintenance : reload_httpd_service]) which happens at the end of the installation can be ignored. You can manually restart the service using service httpd restart.

Ok, I am trying again, thanks for the help. Noticed this during the upgrade -

As long as “…ignoring” is tagged, you don’t have to worry about the error. So in your case, since there is a tag ignoring you can ignore that message.

sudo su, did it. Sorry I missed that, thanks for all help, Ashok

(1) Unfortunately, using the Bahmni installer version `0.86-70` doesn’t fix the problem for me.

I tried to set the log level to debug but I couldn’t even find any reference to port 80 (/var/log/httpd/error_log.1481473749).

After I tried it again to veryfiy that e6fe6 is included in `0.86-70` it worked out one time. This was when I was doing the update while cooking dinner. So my assumption is that the cause of the problem is that there are too little resources available. I have tried to reproduce this multiple times by removing the external monitor, restarting the system, closing all other applications and don’t touching the system during the update. I had two other successful updates and two failed updates with exacly this scenario.

By the way, here is my system information:

  • Ubuntu 16.04
  • 16 GB RAM
  • Intel® Core™ i7-6600U CPU @ 2.60GHz × 4 processor

The VM is set to 2 CPUs, setting it to 4 is discouraged by VirtualBox and also doesn’t make things better if I do it anyway. I have 4096 MB RAM assigned to the VM.

(2) I have logged in with my GitHub accout into go. However, I can only see the build history and not the pipeline definitions. Can you please tell me where to find them and/or grant me read-only access on it?

We fixed it in 0.87 release. It is not available in 0.86. For our CI, pipeline definitions are not available as readonly. You can see the build log here https://ci.mybahmni.org/go/tab/build/detail/single_machine_install/28/defaultStage/1/defaultJob. As well as the whole output of bahmni install command here https://ci.mybahmni.org/go/files/single_machine_install/28/defaultStage/1/defaultJob/bahmni-installer.log.

1 Like