Running UI tests on Travis-CI with Saucelabs

I’ve mentioned it elsewhere already, but wanted to communicate the change more broadly.

We’ve started to run UI tests for Reference Application using Travis-CI and Saucelabs. The setup is described in detail at

The key points are:

  • We are now using docker to startup a server on Travis-CI before running tests (fresh instance for the whole test suite)
  • Tests are executed by Travis-CI
  • Saucelabs is used as a client with a browser driven by tests. Saucelabs connects to the server instance through a tunnel (no access to the test server from the outside world).

The benefits:

  • Our tests are more reliable as no one can interfere with the test server when the test suite is running;
  • If we have a misbehaving test, which leaves some data behind, it will not propagate to the next build.
  • Tests can be run for different branches (but not PRs due to security constraints);
  • Other distributions can copy the setup and have their own distro tested for free (assuming open-source);
  • We have a free account on Saucelabs, which allows us to execute 5 tests in parallel, but we could apply for more to cut the time down. It’s about 20 minutes now for the whole suite, which feels ok.
  • Currently we test only on Firefox 42, but we could easily add more browsers. Again we would have to apply for more agents on Saucelabs so that the build time is acceptable.
  • We can take off the burden of running UI tests from our infrastructure.

Regarding the last point, I would recommend to continue to build and deploy the distro on our CI for manual testing, but skip running UI tests.

One drawback is devs need to pay attention to Travis-CI notifications about broken builds, which is not a big deal given Travis-CI sends an e-mail to a committer when a build is broken. In addition it can be configured to send a bunch more of them.

Eventually I’m envisioning building and deploying the distro for manual testing by Travis-CI as well. We are already building it, but we need to do the deployment part using Docker.


This is great @raff! Both the actual work, and the summary here. :slight_smile: And thanks Soldevelo for your part in this.

Offloading our RefApp UI tests to Travis and Saucelabs sounds sounds very promising. (Even if we don’t do this, we should pursue a similar strategy to free up the very-underutilized int-refapp and int-platform servers.) Some questions on this topic:

  • It’s very important that we have a single place to check he CI status of all the OpenMRS software, so I’d be concerned if we no longer had red/green for the refapp UI tests on Maybe we could set up a plan on our CI that makes a REST call to travis, and passes/fails based on the travis’s last build status. (Given our distributed nature, it’s not enough to count on the dev who made the commit to fix things.)
  • We would want these tests to be triggered by changes to any of the modules in the reference application (at least those with snapshot dependencies), though not on the git commit itself, but after CI has published the snapshot artifact. How would we approach this?
  • Currently, CI builds our refapp artifacts, and Bamboo’s “deployment” feature lets us flexibly put these on different qa/uat/demo environments. It would be bad to lose this.
  • I’d think we should run the tests on both Firefox and Chrome. (Would that turn our ~20 minute plan into a ~40 minute plan or two ~20 minute plans?)
1 Like

I’d love to see a page with all build badges from Travis-CI and Bamboo under, anyone interested in creating one?

My plan is to configure the Distro plan on Bamboo to build the distro and trigger Travis-CI build via REST instead of running UI tests in Bamboo. We will preserve the trigger of UI tests after commits to dependent modules. Not sure how easy it is to write a script to wait for the results of the build from Travis-CI and fail/pass the build on Bamboo. I’ll look into that.

Eventually, I’d like Travis-CI to deploy SNAPSHOTs of modules and trigger the distro build so we can stop using the Bamboo plan.

We can achieve a similar “deployment” feature using Travis-CI by having branches for each environment and pushing changes to them.

Running on both Firefox and Chrome will be one plan, but it shouldn’t double the time, because setting up a server alone (done once for all browsers) takes a few minutes. It’s best to test to see. I’ll try to enable Chrome.

I’d like us to run docker on our int-refapp and int-platform boxes. That way we could use one box to demo work of 3-4 openmrs instances (depending on how much RAM we have on those boxes). We would be able to deploy docker images by simply pushing to dedicated branches.

1 Like

Reporting back that we’re now running tests against Chrome and Firefox, see The build took 24 min 35 sec.

I’ve also configured Bamboo to trigger Travis-CI build instead of running UI tests on Bamboo, see

1 Like

All of this sounds very interesting. I suggest we have a strategy discussion about some of it before you go ahead and do it. :slight_smile:

1 Like

That all looks awesome! Docker and saucelabs are quite powerful :slight_smile:

Just for the fact one can test on IE without creating a windows agent can save the day more than once.

Let me ask you, out of curiosity, why you couldn’t run saucelabs/docker from our Bamboo? I saw there’s a plugin for saucelabs, but I don’t really know what it does exactly

(And yes, I know that creating a travis job is so much easier because it’s text based, but well…)

@cintiadr, I could, but it would take me more time. I guess I’d have to pre-install docker and saucelabs connector for tunneling on all Bamboo agents, then open and close a tunnel when tests are running.

The big benefit of Travis-CI is that the whole CI config is stored in the repo, which means any distro can copy and start using it in a few minutes. Also adding plugins for services like docker, saucelabs, digitalocean, etc. is a matter of adding a few lines to the config.

I think showing a way of building and testing a distro on Travis-C is essential for us to scale up as we cannot afford to create Bamboo plans for other distros. We can’t even build RefApp distro for different openmrs-api versions with every commit, because we would have to wait for a build for too long (we don’t have enough agents).

@darius, I’m planning to start a discussion around all of that after I’m back from holiday.

From here EMRAPI support for OpenMRS 2.0, is not working well because of the recent changes in the reporting module. Therefore, all ui tests should have failed. But on the contrary, the bamboo ci plan for the reference application distro is green. Is this a problem caused by the changes discussed on this thread?

Can you give a link to the build which is running UI tests against Also, I need the build which is deploying openmrs to int02

@cintiadr as you can see here, it no longer runs them on int02 but rather on travis

This one deploys to int02

Oh, right! If I understood it correctly, you are saying that the is green even if TravisCI is red, which would make a lot of sense.

That would make sense, in fact. The Bamboo build only triggers the build in Travis, it doesn’t wait for the response/status. It would be always green, regardless of the status of the build in Travis.

This is a problem!

Can we do a quick fix like creating a new bamboo plan that runs a few times a day and checks the Travis CI build status (via REST?) and passes or fails based on this?

Ultimately, if we’re going to be mixing and matching between our own bamboo, and Travis, we need to have a single dashboard that shows the status of all our builds.

A quick fix could be, when triggering the Travis build, we can wait for the build to finish (I don’t have experience with TravisCI API, but I’d assume it should be easy to have a ‘while build is still running’ bash script).

The other possibility, which doesn’t appear to be terribly hard, would be migrate the soucelabs to Bamboo.

It’s all a matter of what do we prefer, keep the build in Bamboo, spread them into two CIs, or if we plan on eventually migrate things to travis.

I’ve added for tracking the travis-ci build on Bamboo as the quick fix. It is checked every 15 minutes.

@raff this new plan is failing. Is it because of the reference app build failure? Or something else?

Added an info to the logs, see

@raff with this new approach, do you have any pointers regarding how to tell the commit which leads to a failure?

@dkayiwa, go to as bamboo logs suggest and see build message e.g. “Triggered by” You can then go back to Bamboo and investigate, if it was triggered by the commit to distro or any of its modules.

@raff and @adamg could any of you be knowing why this is failing?

@dkayiwa, see e.g.

It seems like the logout link got broken and throws 404.

Also digging further it started to fail at, which leads to