We are now using docker to startup a server on Travis-CI before running tests (fresh instance for the whole test suite)
Tests are executed by Travis-CI
Saucelabs is used as a client with a browser driven by tests. Saucelabs connects to the server instance through a tunnel (no access to the test server from the outside world).
The benefits:
Our tests are more reliable as no one can interfere with the test server when the test suite is running;
If we have a misbehaving test, which leaves some data behind, it will not propagate to the next build.
Tests can be run for different branches (but not PRs due to security constraints);
Other distributions can copy the setup and have their own distro tested for free (assuming open-source);
We have a free account on Saucelabs, which allows us to execute 5 tests in parallel, but we could apply for more to cut the time down. It’s about 20 minutes now for the whole suite, which feels ok.
Currently we test only on Firefox 42, but we could easily add more browsers. Again we would have to apply for more agents on Saucelabs so that the build time is acceptable.
We can take off the burden of running UI tests from our infrastructure.
Regarding the last point, I would recommend to continue to build and deploy the distro on our CI for manual testing, but skip running UI tests.
One drawback is devs need to pay attention to Travis-CI notifications about broken builds, which is not a big deal given Travis-CI sends an e-mail to a committer when a build is broken. In addition it can be configured to send a bunch more of them.
Eventually I’m envisioning building and deploying the distro for manual testing by Travis-CI as well. We are already building it, but we need to do the deployment part using Docker.
This is great @raff! Both the actual work, and the summary here. And thanks Soldevelo for your part in this.
Offloading our RefApp UI tests to Travis and Saucelabs sounds sounds very promising. (Even if we don’t do this, we should pursue a similar strategy to free up the very-underutilized int-refapp and int-platform servers.) Some questions on this topic:
It’s very important that we have a single place to check he CI status of all the OpenMRS software, so I’d be concerned if we no longer had red/green for the refapp UI tests on ci.openmrs.org. Maybe we could set up a plan on our CI that makes a REST call to travis, and passes/fails based on the travis’s last build status. (Given our distributed nature, it’s not enough to count on the dev who made the commit to fix things.)
We would want these tests to be triggered by changes to any of the modules in the reference application (at least those with snapshot dependencies), though not on the git commit itself, but after CI has published the snapshot artifact. How would we approach this?
Currently, CI builds our refapp artifacts, and Bamboo’s “deployment” feature lets us flexibly put these on different qa/uat/demo environments. It would be bad to lose this.
I’d think we should run the tests on both Firefox and Chrome. (Would that turn our ~20 minute plan into a ~40 minute plan or two ~20 minute plans?)
I’d love to see a page with all build badges from Travis-CI and Bamboo under http://om.rs/ci, anyone interested in creating one?
My plan is to configure the Distro plan on Bamboo to build the distro and trigger Travis-CI build via REST instead of running UI tests in Bamboo. We will preserve the trigger of UI tests after commits to dependent modules. Not sure how easy it is to write a script to wait for the results of the build from Travis-CI and fail/pass the build on Bamboo. I’ll look into that.
Eventually, I’d like Travis-CI to deploy SNAPSHOTs of modules and trigger the distro build so we can stop using the Bamboo plan.
We can achieve a similar “deployment” feature using Travis-CI by having branches for each environment and pushing changes to them.
Running on both Firefox and Chrome will be one plan, but it shouldn’t double the time, because setting up a server alone (done once for all browsers) takes a few minutes. It’s best to test to see. I’ll try to enable Chrome.
I’d like us to run docker on our int-refapp and int-platform boxes. That way we could use one box to demo work of 3-4 openmrs instances (depending on how much RAM we have on those boxes). We would be able to deploy docker images by simply pushing to dedicated branches.
@cintiadr, I could, but it would take me more time. I guess I’d have to pre-install docker and saucelabs connector for tunneling on all Bamboo agents, then open and close a tunnel when tests are running.
The big benefit of Travis-CI is that the whole CI config is stored in the repo, which means any distro can copy and start using it in a few minutes. Also adding plugins for services like docker, saucelabs, digitalocean, etc. is a matter of adding a few lines to the config.
I think showing a way of building and testing a distro on Travis-C is essential for us to scale up as we cannot afford to create Bamboo plans for other distros. We can’t even build RefApp distro for different openmrs-api versions with every commit, because we would have to wait for a build for too long (we don’t have enough agents).
@darius, I’m planning to start a discussion around all of that after I’m back from holiday.
From here EMRAPI support for OpenMRS 2.0, http://int02.openmrs.org/ is not working well because of the recent changes in the reporting module. Therefore, all ui tests should have failed. But on the contrary, the bamboo ci plan for the reference application distro is green. Is this a problem caused by the changes discussed on this thread?
That would make sense, in fact. The Bamboo build only triggers the build in Travis, it doesn’t wait for the response/status. It would be always green, regardless of the status of the build in Travis.
Can we do a quick fix like creating a new bamboo plan that runs a few times a day and checks the Travis CI build status (via REST?) and passes or fails based on this?
Ultimately, if we’re going to be mixing and matching between our own bamboo, and Travis, we need to have a single dashboard that shows the status of all our builds.
A quick fix could be, when triggering the Travis build, we can wait for the build to finish (I don’t have experience with TravisCI API, but I’d assume it should be easy to have a ‘while build is still running’ bash script).
The other possibility, which doesn’t appear to be terribly hard, would be migrate the soucelabs to Bamboo.
It’s all a matter of what do we prefer, keep the build in Bamboo, spread them into two CIs, or if we plan on eventually migrate things to travis.
And thanks Soldevelo for your part in this.