Rebuilding mdsbuilder

@darius, @wyclif, and @dkayiwa:

Following up from yesterday’s design discussion that ended up focusing on mdsbuilder. A asked the infra team and @cintiadr pointed out mdsbuilder is built manually per the OpenMRS Environments wiki page:

From the SDK documentation, I’m assuming this means that we’d need to update openmrs-distro.properties, download it locally, run the appropriate SDK command to build the distro, and then – assuming that produces a docker image – push the docker image to openmrs/openmrs-distro-mdsbuilder on Docker Hub.

Currently, I only see myself, R0bby, Rafał, Cintia, OpenMRS CI, and Darius as having access on Docker Hub.

If we don’t already have one, it seems like a CI plan for deploying mdsbuilder would be better. Then anyone could update openmrs-distro.properties and trigger a rebuild in CI.

I’m also assuming we could also come up with a better approach to updating the CIEL dictionary on mdsbuilder than this (requiring SSH access to mdsbuilder):

  1. Copy latest version of CIEL’s zip file onto mdsbuilder machine

     scp /path/to/ciel.sql.zip username@mdsbuilder.openmrs.org:/home/username/
    
  2. Get access to shell on mdsbuilder

     ssh username@mdsbuilder.openmrs.org (currently burke, cintia, raff... need to add release manager)
    
  3. Unzip CIEL SQL file and put it into /root/docker/mdsbuilder/dbdump/ folder on host (will be accessible to MySQL)

     cd
     unzip ciel.sql.zip
     sudo mv cel.sql /root/docker/mdsbuilder/dbdump/
    
  4. Log into database docker container and enter msyql

     sudo docker exec -it mdsbuilder_db_1 bash
     mysql -u $MYSQL_USER -p$MYSQL_PASSWORD $MYSQL_DATABASE
    
  5. From MySQL prompt within the mdsbuilder database container, source the CIEL sql file

     source /docker-entrypoint-initdb.d/ciel.sql

I see an Update CIEL on mdsbuilder plan in CI that @raff had set up. It looks like it was intending to download a SQL zip directly from dropbox and apply it via MySQL, but the plan is disabled and it’s hard for me to see how the one “mysql” command in it would accomplish anything.

But I think this is the right direction. From what I can see, we would benefit from two CI tasks:

  1. A manually triggered task that uses the SDK to build an instance of mdsbuilder from mdsbuilder/openmrs-distro.properties and push it to Docker Hub. I don’t know if @cintiadr would support configuring mdsbuilder to auto-update when docker hub is updated (e.g., using something like watchtower) or stick with a manual step of someone on the infra team re-building mdsbuilder once the updated image had been deployed to Docker Hub.

  2. A manually triggered task that fetches the correct CIEL dump (based on a variable or repo file) and deploys it to mdsbuilder.

I used to manually update CIEL on the mdsbuilder server – before all the server improvements and migration. OpenMRS was deployed or updated automagically. I’ve never seen the ci plan for that. When a new CIEL dictionary was release, I would manually source the mysql dump.

It would be helpful to restore the mdsbuilder server and automate the CIEL dictionary update.

Ellen

1 Like

(I’m not going to engage any further on the technical side of this mdsbuilder issue, since I have no time. Thanks for digging into this, Burke and others.)

Based on this thread, and on our recent work around reviving the Ebola demo server, it seems like it would be super helpful if there was a semi-automated way (even manual scripts are fine, or just a wiki page describing the steps) to say, for X hosted server:

  1. Install it or update it to what’s defined in an openmrs-distro.properties file
  2. Apply a sqldump to it (either a complete DB like demo data, or a partial DB like the CIEL distributions)

Burke, I haven’t got a chance to complete the CI plan for updating CIEL on mdsbuilder.

Alternately, when building a new mdsbuilder image, I would include the CIEL script inside and modify the server startup script to load it (via a new SDK param similar to -DdbSql). This way you would only have to pull the latest docker image and run it on the mdsbuilder machine.

Yes, that would be ideal! :smiley:

We have two ways of doing that. One is asynchronous (we create a dockerhub webhook on push, but the there’s no feedback anywhere if it was successful or not) or synchronous (bamboo will call for a new deploy, update all images and wait for the new containers to be healthy). We are using both ways already.


I suppose there are a lot of options to apply the sql.

One option would be the sql file commited in dbdump. It would require an ansible run (and new volumes). Other option is to actually run that locally. There’s nothing requiring a server to be running to run docker.

The other option is to create a docker service in the docker-compose file that will run the mysql commands when started. So in Bamboo you create the docker image with the desired SQLs, deploy it, and they will run.

The third option is what @raff suggested, that the mdsbuilder image itself might do it during initialisation.