Update on 2.x E2E tests

Hello there,

Today, I had a productive call with @kdaud to clarify various aspects of the openmrs-contrib-qaframework, including its structure, failing tests, progress on 2.x E2E, pending work, and implementation issues.

Before the call, I familiarized myself with the QA Framework technical documentation and two helpful YouTube videos[1] [2]. During the call, Daud expertly explained the project’s directory structure and test configurations, including the transition from legacy to BDD tests.

Here are my key takeaways from the discussion:

  1. The bamboo integration is not working properly and has been disabled due to resource consumption on the CI server, and @raff has been asked to look into it.
  2. Critical functionalities of 2.x have been covered with E2E tests
  3. Tests are failing randomly, but not due to outdated tests but rather issues with running them on Github actions

Going forward, I have identified the following action items:

  1. Get the older PRs merged - @kdaud
  2. Check the status of bamboo integration resolution with @raff
  3. Resolve random test failures on GitHub - @piumal1999

@raff, could you please give us an update on the status of the bamboo integration resolution? I couldn’t access the plan as the bamboo dashboard seems to be broken.

If you have any questions or comments, please feel free to share them below. Thank you!

References:

  1. OpenMRS QaFramework - YouTube
  2. Technical Workflow of OpenMRS QaFramework - YouTube
  3. QA Framework: Technical documentation (Cucumber-Selenium & Cucumber-Cypress Set up) - Documentation - OpenMRS Wiki
  4. Broken QA builds
  5. Slack conversation regarding the resource consumption

cc: @raff @dkayiwa @grace @ibacher @mherman22 @piumal1999 @jnsereko @anjisvj

7 Likes

Thanks so much @jayasanka for this clear update!!

Weird that bamboo is completely down - @burke or @dkayiwa are you able to restart it?

@raff looking forward to hearing your update :slight_smile:

Bamboo was running out of disk space due to an issue with backups not getting processed as expected, @ibacher was able to get it running, and @cintiadr mentioned she’d take a look at the backup process issue when she gets a chance. So Bamboo (ci.openmrs.org) should be running again and hopefully not going down again.

Ultimately, I think the backup backlog was caused by the Jetstream networking errors on Jan 27th. Hopefully that’s a one-off. Realistically, if this was the only fallout from that, I think this shows the overall robustness of our infrastructure.

Thanks for the update @burke and @ibacher. I’m looking forward to hearing an update from @raff regarding the 2.x E2E integration.

https://openmrs.slack.com/archives/CP343JH1R/p1662728012364569

@jayasanka I’m sorry it took me a while to get back to this. I’ve made necessary changes to Reference Application - Distribution 3.x E2E: Plan summary - OpenMRS Bamboo

However, I discovered that there’s an issue with setting up the 3.x from scratch. Some module seems to be blocking startup procedure. I’m investigating that right now. I was able to reproduce the issue locally as well. I’ll update you once I find the fix.

1 Like

Thanks for the update @raff !

It appears that the 2.x related plans are not including any tests.

  1. Reference Application - Distribution 2.x: Plan summary - OpenMRS Bamboo
  2. Reference Application - Refapp 2.12.0-dev: Plan summary - OpenMRS Bamboo

How can we resolve this?

Also, have discovered the plan CONTRIB-QA, where 2.x E2E tests were configured. It seems it is triggered when there’s a commit to the GitHub - openmrs/openmrs-contrib-qaframework repo. As far as I know, the job is disabled due to resource consumption on the CI server.

I think it would be better if we could eliminate the CONTRIB-QA plan and integrate the 2.x E2E tests into the 2.x platform and refapp plans.

What do you think?

@raff just to be clear… you’re using Distribution 3.x as the “mothership”/official plan, right? this is not actually the build we are using for O3 (the 3.x E2E one was a temporary clone for purpose of practicing setting up automated tests within our ci); the one to use is Reference Application - Distribution 3.x: Plan summary - OpenMRS Bamboo - this is the one that builds dev3 and is triggered by GitHub actions.

  • Distribution 3.x: Main plan, will remain the plan, that builds & deploys to dev3
  • Distrubution 3.x E2E: Exists so we can try out integrating the steps that run the tests as part of the build. We did this as a cloned plan b/c we didn’t want to stop deployments while we figured out the E2E tests (e.g. right now it’s currently failing, which would have meant nothing would be deploying to dev3). Ideally we want these tests running on the official Distribution 3.x build (then we’d remove this cloned plan entirely) and per @jayasanka today we are almost ready to do this.
  • Distribution 3.x Releases: New, separate plan @ibacher recently created, designed for handling O3 releases. Shouldn’t be manually run; will only execute when we intentionally kick off a formal O3 RefApp Release. E.g. see Ian’s note here from today about a new O3 release version.

image

Thanks @raff this is extremely helpful to know - actually it might explain some of the issues @moduguvikram has been running into with trying to setup the O3 RefApp. I’ve filed an issue in our new “Operation Stabilize” epic for stabilizing the O3 RefApp; assigned this issue to you Raff :slight_smile: : [O3-1864] - OpenMRS Issues

I merged some and following up on the pending ones with review comments.

The module is configured at a specific time to auto-trigger a plan every after 24 hours. However, any push done on the module by default auto triggers a ci plan.

@jayasanka what could be the ideal of having the tests integrated into two modules?

It can be added as a new Test stage to Reference Application - Distribution 3.x: Plan summary - OpenMRS Bamboo after the Build stage.@jayasanka please let me know if you want me to do that or you want to try it yourself first.

I feel that it should be part of Reference Application - Distribution 3.x: Plan summary - OpenMRS Bamboo similar to OpenMRS Core - OpenMRS Core 2.5.x 162: Build result summary - OpenMRS Bamboo where we have the Release stage, which is triggered manually. As part of the Release stage we would create a tag in git and dockerhub. We were also discussing preparing a release on a separate release branch so that it does not affect the ongoing development. We could deploy that branch to test3.openmrs.org and when ready to do a release, tag and deploy to demo.openmrs.org.

For right now, the workflow with having two jobs is more maintainable… The central problem is that we need to keep dev3 up-to-date with the latest changes to any of the frontends. We do that by building against the next tag from NPM, ensuring that we’re always pulling in the latest versions. However, for releases, we need fixed version numbers so that the release builds are repeatable. The easiest way to do this is via two commits, one which updates things to fixed version, the other which restores the next tags. In any case, at the present moment, it does need to be a manual choice about which versions are being included in a release, so having this workflow saves that.

I also don’t see any clear advantages at this point and at the current level of maturity to trying to segregate the releases to a separate branch. This is because most of the churn in the repo, right now, is updating the app configuration, which are commits that we’d then need to also do to the release branch (but, again, not every commit would necessarily be an appropriate candidate for merging into a release branch, e.g. if we were to add a new frontend module to the 3.x app, which happens from time-to-time, we wouldn’t necessarily want that in the release…).