CI within OpenMRS - Bamboo vs Travis vs GitHub Actions vs Azure Pipelines

[ibacher (Ian Bacher)]

@burke

GitHub Actions – some groups are experimenting with these as an alternative to Travis; however, these are limited to 2000 minutes/month [ref].

So ref, but for public repos, GitHub Actions are currently free. The 2000 minutes limit applies to just private repos.

@burke

For devs, are there reasons other than reliability to avoid using our Bamboo CI?

I’m a bit less concerned about mystery failures, but I’ve got a few other concerns:

1. We’ve tended to use Travis for PR-level checks where the main concern is to ensure we’ve built, tested and generate code coverage data for a PR before reviewing it. I’m sure there’s some way to configure Bamboo to run on PRs, but we’re not currently doing that.
2. There are only 3 builder runners configured for Bamboo. For this most part, this is probably the right number, but during periods where we are doing a large number of module releases, builds will invariably queue up. This might also be more of an issue if we use Bamboo to address #1 above.
3. We have a wide-range of build environment needs from, e.g., MFs (pure NodeJS, but needing relatively up-to-date versions of Node), standard modules (pure Maven on Java, but would be nice to be able to test a number of Java versions), to UI 2.0 focused modules (generally needing both Java and somewhat older versions of Node). I’m not sure our volunteer infrastructure team has the time to be able to maintain this successfully.

@burke

I believe the draw of Travis or GitHub Actions is being able to manage them using GitHub credentials, their reliability, and the ease of creating them.

For me, it comes down to several features that make these preferable to Bamboo, at least as we’re currently using them:

1. I can trigger builds from PRs rather than waiting for code to be committed. The status of the build gets added to the PR and I can update code coverage tools, etc. before merging a request in.
2. Matrix-style builds where I can, e.g., specify multiple Java versions and build against all of them.
3. Support for different OSes. One of the main reasons I moved the FHIR2 module to GitHub Actions over Travis was easy support for building on Windows when we ran across a few bugs that only appeared on Windows.
4. Support for a range of environments that are impractical to support in a non-virtualised way.
5. The build configuration lives with the code (e.g., .travis.yml or .github/workflows/*.yml). This means not only can I easily see the steps that the build will follow (this information is visible in Bamboo, but it takes substantially more clicking to get there), but I can easily make changes to the build and if that causes the build to break, I can see what changed in one place.
6. Familiarity and ecosystem. Many OSS projects build on Travis and GitHub actions. This means that developers from outside OpenMRS are more likely to have some familiarity with these tools.

@burke

Lastly, is there any way for us to estimate our CI build usage (e.g., minutes/month)? I realize our builds are mostly happening across Travis-CI (.org ± .com) and Bamboo at the moment. Does anyone know if there’s a way to estimate our typical build time in minutes/month from either of these environments?

I tried to get some numbers for Travis in this post, but Travis doesn’t keep information around for long (and, of course, we’re actually not using any time on travis-ci.com anymore).