CI within OpenMRS - Bamboo vs Travis vs GitHub Actions vs Azure Pipelines

Development
Tags: #<Tag:0x00007f5ff9c405f8>
#1

Squads are running into issues with builds now that Travis has put limits on number of builds.

From what I’ve heard:

  • Bamboo – self-hosted with unlimited usage and provides a centralized dashboard, but not trusted as reliable as builds sometime fail for mysterious reasons
  • Travis – was popular, but now applying monthly limit [ref]
  • GitHub Actions – some groups are experimenting with these as an alternative to Travis; however, these are limited to 2000 minutes/month [ref].
  • Azure Pipelines – allows 1800 minutes/month (maybe unlimited… wording isn’t exactly clear) for open source with unlimited for self-hosted CD/CI [ref]

I believe @jdick & @ibacher have written to Travis asking for OSS credits; I’ve just emailed Travis support from our infrastructure email account as well, hoping we might be able to get OSS credits and avoid having to shift all Travis builds elsewhere.

I believe the draw of Travis or GitHub Actions is being able to manage them using GitHub credentials, their reliability, and the ease of creating them.

For devs, are there reasons other than reliability to avoid using our Bamboo CI?

@cintiadr, I believe the reliability issues on Bamboo are primarily due to (1) depending on jobs to clean up after themselves and (2) caches filling up disk space. Would it be possible to improve Bamboo reliability by isolating these builds (via Docker and/or by automatically building & destroying Bamboo agents)?

Lastly, is there any way for us to estimate our CI build usage (e.g., minutes/month)? I realize our builds are mostly happening across Travis-CI (.org ± .com) and Bamboo at the moment. Does anyone know if there’s a way to estimate our typical build time in minutes/month from either of these environments?

/cc @cintiadr @dkayiwa @jennifer

2 Likes
CI Jobs fails on openmrs-distro-referenceapplication-uitests module
#2

Bamboo is not available for implementations, would be great if more implementations can leverage it, hence they move to whatever the experience they can find within their organizations

#3

So ref, but for public repos, GitHub Actions are currently free. The 2000 minutes limit applies to just private repos.

I’m a bit less concerned about mystery failures, but I’ve got a few other concerns:

  1. We’ve tended to use Travis for PR-level checks where the main concern is to ensure we’ve built, tested and generate code coverage data for a PR before reviewing it. I’m sure there’s some way to configure Bamboo to run on PRs, but we’re not currently doing that.
  2. There are only 3 builder runners configured for Bamboo. For this most part, this is probably the right number, but during periods where we are doing a large number of module releases, builds will invariably queue up. This might also be more of an issue if we use Bamboo to address #1 above.
  3. We have a wide-range of build environment needs from, e.g., MFs (pure NodeJS, but needing relatively up-to-date versions of Node), standard modules (pure Maven on Java, but would be nice to be able to test a number of Java versions), to UI 2.0 focused modules (generally needing both Java and somewhat older versions of Node). I’m not sure our volunteer infrastructure team has the time to be able to maintain this successfully.

For me, it comes down to several features that make these preferable to Bamboo, at least as we’re currently using them:

  1. I can trigger builds from PRs rather than waiting for code to be committed. The status of the build gets added to the PR and I can update code coverage tools, etc. before merging a request in.
  2. Matrix-style builds where I can, e.g., specify multiple Java versions and build against all of them.
  3. Support for different OSes. One of the main reasons I moved the FHIR2 module to GitHub Actions over Travis was easy support for building on Windows when we ran across a few bugs that only appeared on Windows.
  4. Support for a range of environments that are impractical to support in a non-virtualised way.
  5. The build configuration lives with the code (e.g., .travis.yml or .github/workflows/*.yml). This means not only can I easily see the steps that the build will follow (this information is visible in Bamboo, but it takes substantially more clicking to get there), but I can easily make changes to the build and if that causes the build to break, I can see what changed in one place.
  6. Familiarity and ecosystem. Many OSS projects build on Travis and GitHub actions. This means that developers from outside OpenMRS are more likely to have some familiarity with these tools.

I tried to get some numbers for Travis in this post, but Travis doesn’t keep information around for long (and, of course, we’re actually not using any time on travis-ci.com anymore).

2 Likes
#4

Ah. So, it sounds like any squads running into Travis limits could migrate their Travis builds to GitHub Actions and we could continue using Bamboo for key community artifacts/packaging/releases.

2 Likes
#5

Nice. That sounds good to me. FYI @jdick

#6

There’s a somewhat separate but related issue: travis-ci.org shuts down on December 31st [ref]. This affects the vast majority of Travis builds we are currently running. Given the lack of clarity that Travis can provide around their plans for supporting OSS, are there any objections to starting to migrate OpenMRS modules using Travis to GitHub actions?

To be clear, I’m only talking about migrating repositories which are standard OpenMRS modules (so, setting aside OpenMRS infrastructure related projects and the MF stuff) and only those that are clearly not relying on, e.g., Travis secrets and, again, only for PRs and commits to the main branch. I’ll still leave releases, etc. for Bamboo.

#7

No objections from me. :slight_smile:

#8

I’m hearing from Twitter that they are currently not granting any credits right now as management is deciding a few things. Hopefully they will approve us in January, but I wouldn’t expect this month.

Yeah, that’s the cache. It’s mostly us being mean to ourselves. Docker builds are available, but I never used them. Also, there’s monitoring when we are running out of disk, but I don’t have others actively looking.

But the main reason we went with travis and bamboo is because travis can be ran in forks, something I don’t think Bamboo is capable of. Moving our releases and all the miscellaneous builds will be a little bit of an adventure. I’m not sure that’s actually worth doing at all. But there are lot of builds that can be easily moved without too much harm or work.

I’m fairly sure we can get that from agents usage in Bamboo, but probably require a database query or something like a plugin.

I attempted to add more, but it’s hard to justify it now, really.

Technically, the docker builds are supposed to be there to address that, but there was never really someone driving it (and I didn’t want to complicate the builds more than people would be willing to). I also can always add nvm or similar tools, which is usually good enough. That said, maybe moving those builds to cloud CIs might be just easier.

I think there a lot of builds that would benefit from moving from Bamboo to a cloud CI, but we just have to be really careful transporting their releases. I believe we will still want to keep bamboo for the time being, and we will learn a few things that weren’t easy to do in a cloud CI.

2 Likes
#9

We also never started using bamboo’s version of CI as code, same way.

#10

Personally, I’ve got no desire to move stuff that’s working on Bamboo somewhere else. If nothing else, it’s nice to have one place to check build status rather than having to look at 40 different repositories (that’s the one thing we lose by moving away from Travis… there’s a dashboard for the whole organization).

Yeah, the email I got from Travis is identical to the emails I’ve seen on Twitter (“We are awaiting further directives from management regarding the soon-to-be-released OSS/Education plan”). I’m not really holding my breath…

Sounds like it would solve most of the issues I raised :smile:, but that does sound like a big effort to get going.

1 Like
#11

I’ve been making some progress with this. For those who are interested, I’ve started this conversion for several OpenMRS repos. I’m basically setting them up with this, plus or minus a few variations. The script I’m using to do this is this (with some manual variations). I’ve been avoiding any repositories that have a setup beyond just some relatively straight-forward Maven commands, as those will require more effort and, in all probability, generating new secrets to migrate them. The repositories that have been converted so far can generally be found through this GitHub query.

4 Likes
#12

Thank you @ibacher for your work on this! This has been in the back of my mind when I heard of GitHub actions since I have seen build times increasing for our PR builds on Travis. Also since GitHub actions are so close to the code/access to GitHub APIs (GitHub script). Plus when we had some tests only failing for devs on Windows it did not take me long to run them on PRs on Windows using GitHub actions.

So please let me know if I can support you in that effort :grin:

1 Like
#13

Thanks @ibacher for this move. Any update on this and please let me know where I can support you in this effort

1 Like
#14
#15

is this per project/user or what! this seems a limit still

#16

As I noted above, the limit only applies to private repos. At least for now, GitHub actions is free for public repos. In any case, it’s a per-repo limit.

I’ve converted quite a few repos over to using GitHub actions already. You can find the list of unconverted repos through this GitHub search and we should probably target repos based on this listing.

I generally only converted repos that used a very simple Travis file, generally the default Java build, though I can see a couple I missed. The remaining simple builds can be converted using this script. For build that aren’t simple, they’ll probably need more care to understand what the build is doing and how to replicate it.

3 Likes
#17

I have also written one that compiles uitestframework, distro and runs qaframework e2e tests at:

A travis equivalent for this is available at: openmrs-contrib-qaframework/.travis.yml at master · openmrs/openmrs-contrib-qaframework · GitHub

3 Likes
#18

@k.joseph does this mean that QAFramework PRs will now auto-run via GitHub Actions rather than Travis?

1 Like
#19

We could keep both running until travis finally closes. but i think you are raising a good point, we can disable travis

1 Like
#20

Upcoming call to review remaining unconverted repos

FYI to everyone following this thread… in 1 week, we’re having a get-together on next week’s Monday design time to go through the unconverted repos and double-check for critical ones that need to be moved from Travis to GitHub Actions. (Per discussion on the TAC last week.)

When: Mon March 29, at 8pm EAT / 7pm CET / 5pm UTC / 1pm EDT / 10am PDT

Where: https://om.rs/zoomdesign

GCal link here

2 Likes
2021-03-09: Session to Review Travis Repos