Automating build targets with Docker or other tools

Today (2015-06-11) in the Developers Forum, the topic of creating ephemeral build targets (rather than dedicated full-time environments) came up. One approach would be to use automation tools like Docker to launch and destroy these as they are needed.

Let’s discuss in more detail the requirements of this type of approach, and the pros and cons of some of the potential solutions, so we can evaluate moving forward with such an approach!

I’m not sure what we want to achieve in here, and I think we need more context to understand exactly which tools would help more. What do we want to test? Advantages and disadvantages depends pretty much what we are comparing to. Do we want to only have a disposable tomcat+database or do we want to test running the full machine?

Advantages of ephemeral build targets I can think of are: isolation; more types of tests (running across different platforms, without asking for more and more VMs). It’s self service. Disadvantages: more time to run each build, more work for OPS (more monitoring, more things installed on agents or whatever). One won’t be able to login to understand the problem, so some automatic screenshots might be needed, as well tomcat log collection and all the possible data one would use to investigate a failure.

I think I mentioned on the MariaDB thread/issue, there are several options between getting a dedicated environment (like dev01/dev02) and get everything running inside the agent (tomcat&database). Including some virtualisation, like Vagrant + virtualbox (if the build agents are not virtualised themselves), or even amazon ec2 instances that you can start and destroy (vagrant + aws is a valid option too). Dockerise the full application might be an option too, if somehow that’s desirable.

–//–

I believe Docker is quite suitable for the task of a disposable container, if the purpose is doing more real-life database tests. I’ve installed docker on the Bamboo agents recently; newer versions of Bamboo have a Docker task (it’s just a nicer wrapper for the executable, but it might be helpful if we decide to go that path). I’m ready to test that solution, I just would like someone to explain to me how to set up the database for OpenMRS before running the tests. Please, feel free to mess around with https://ci.openmrs.org/browse/RELEASE-MDT

It takes a little bit of docker tuning to get things in a good shape, as we could possibly have two docker builds on the same machine at the same time as we have two agents (and we don’t want one to interfere with the other). We might overload the agent on CPU/Memory/IO, we truly need to keep an eye on its metrics.

Disadvantages with Docker is that you cannot access the destroyed container to understand the problem. Security is not an issue on this case. Advantages of having the database on the agent itself would be that we don’t need to monitor the agents to make sure the database is running there; and of course, isolation. In case someone wonders, it’s possible to bring tomcat up using cargo maven plugin, so it shouldn’t be extremely hard to setup.

1 Like

Filling in some of the details that @michael left out…

We keep coming up with different scenarios that would be nice to be able to run our existing UI test suite against. Recently (in addition to what we’re already doing):

  • mariadb
  • (Eventually postgresql)
  • (for each supported database…) installing OpenMRS + modules from scratch
  • java 8 [this one just came up on the dev forum today]
  • (maybe other JVMs)

It seems that we don’t actually want to have a dedicated VM for each of these, that sits idle 95% of the time. I asked the infrastructure team if we are set up to handle these transient environments, but we were in the last minute of the call, so we decided to move this to talk.

So, if we were taking the docker route, the idea here would be to have a variety of docker containers (e.g. mysql, mariadb, postgresql, tomcat 7 running on java 7, tomcat 8 running on java 8), and we’d compose these together into the various combinations that we actually want to test.

(I’m sure we could do the equivalent thing with vagrant.)

My underlying thought process is “we have a set of UI tests that test the application end-to-end, so it should be trivial to test these against one more environment (as long as we’re not doing it on every commit).” Is there something wrong with that way of thinking? (Of course we could be more deliberate and have more targeted tests for each scenario, but my thought is that the bottleneck today is “QA thinking” capacity, and ephemeral virtual environments should be cheap.

1 Like

Docker is made in a way that you should only run one process on a container. So, one container for the database, another for the web application. You could potentially have multiple process, but things are not really designed that way (it gets harder to check logs, if app is up, and so on).

Anyway, I cannot think right now of any advantage of running tomcat + application on a docker container to run the end-to-end tests, instead of just using cargo maven plugin to bring tomcat up and down. We’d have the make sure we are able to retrieve the all the logs from the docker container, and ‘deploying’ the application to a docker container (mounting a folder) is not really what we have in production. It would be a different environment anyway.

We can try to see if it gets too complicated. A good outcome would be to have docker containers you can run locally on development. Even if we don’t actually run docker on build, this - by itself - might be enough to create docker containers for QA purposes, evaluation or whatever.

For unit and other integration tests, it doesn’t matter, you’d need to run on the agent anyway.

About the ‘installing OpenMRS + modules from scratch’ bit, I think it truly depends of how rigorous we want to go. If we say ‘we want to test that from a vanilla ubuntu, I can run this puppet manifest and I willl get openmrs running on a certain database’. That would test the puppet manifest itself on ubuntu, not openmrs. No functional tests would be likely to be picked up only on this scenario, right? While it’s not a bad idea at all, it just feels it’s different of running openmrs in a different database, jdk or tomcat. I don’t have a lot of context, which kind of deployment problems OpenMRS usually has?

Even if you decide to create more vagrant boxes or docker containers for the applications, I think getting docker containers for database is the first step. If we dislike docker badly, we can always decide to install databases ‘somewhere’.

This all makes sense (i.e. docker containers for DBs are good, but containers for tomcat + application aren’t actually helpful).

If we really cared about being most production-like, it seems that a vagrant VM with both OpenMRS and the DB is most realistic. But I don’t think this is really a concern.

By “from scratch” I meant that assuming you already have tomcat + empty db, then applying openmrs-core WAR + refapp modules on top of that installs successfully (creates all metadata, shows the login screen, +/- running the ui tests).

The reason I particularly want this is that there was a point in the past where you could install OpenMRS on PostgreSQL (thanks to lots of tedious work by @sunbiz) but because nobody was paying attention, we lost this. I don’t want that to ever happen again.

1 Like

In other news:

http://blogs.atlassian.com/2015/06/docker-containers-bamboo-winning-continuous-delivery/

Yes, Bamboo 5.8 brings docker tasks. If we decide to start using docker from Bamboo, it’s quite handy.

But the dockerized bamboo remote agents don’t exactly solve our problem, unless we decide to create our own docker containers with everything we need installed.

1 Like

I am working on doing this for the dashboard currently using compose.

1 Like