Over the last week, I attempted to evaluate our automation technical pipeline, debt, and opportunities. There, I focused on E2E tests because we already have mechanisms in place for unit and integration testing, but E2E tests aren’t well-established yet. Another reason is that, when considering the openMRS architecture, E2E tests provide more accurate feedback than unit and integration tests because they run in a full real-life environment rather than a mock environment.
I reviewed our current approach as well as the documentation of other product teams on E2E test automation, such as Hackney, Wix, and Single SPA, to determine what might be useful for the OMRS application. There, I discovered the following flaws in the current workflow.
E2E tests are not a part of the devs’ workflow.
The developer engagement is extremely low with the current setup. One reason would be that we aren’t providing any immediate feedback for developers. Therefore developers don’t have an idea even if their changes break tests; i.e. break the product. There are countless cases where we have identified critical breakdowns in components weeks after PRs have been merged.
Unclear Guidance
Best practices, and our expectations around testing, are not clearly documented.
Cucumber is slowing down O3 test automation
What Gherkin syntax is really good for is if we have a less technical team to implement tests. You can have a team of developers create a library of steps and then have less technical users write the scenarios, which is not how we work. Everything we’re getting with Cucumber can be done with a good naming convention for tests and having tests properly organised into suites. Making sure that non-technical users can read a report of tests and understand what parts failed; is a crucial component of effective test design. In O3 we use a separate library to integrate cucumber with cypress to have a bridge we can maintain. In order to achieve that, we are sacrificing some cypress fx, especially as cypress upgrades. Those libraries are not well maintained.
Yesterday, @ibacher , @dennis , @jnsereko and I had a call and discussed the above further and identified the following as the next steps:
Fix failing tests and enable tests in the deployment pipeline
We already have a workflow to run E2E tests on deployment. In order to get a better result, first we need to make sure all the tests are up-to-date with the current implementation.
Improve DevX
In order to make the E2E framework sustainable we need to improve the developer experience.
Evaluate the current tools and technologies we use for E2E testing.
Provide immediate feedback to developers
Store tests in a convenient place
The farther away the tests from the codebase are, the fewer attention developers pay to it. We need to make sure tests are placed in a place where developers can easily update the test with their changes. One possible solution would be to have tests in each micro frontend repository itself. So that developers can easily update tests along with their PRs. It is how wix.com address this issue.
FWIW, with openmrs-core 2.6.0, we have introduced Testcontainers such that one is able to run all existing tests on MySQL with: mvn test -DuseInMemoryDatabase=false -Ddatabase=mysql or on PostgreSQL with: mvn test -DuseInMemoryDatabase=false -Ddatabase=postgres
As part of the next steps, do you also plan to improve the documentation?
This is one of the best paragraphs that you have ever written for my consumption!!! Ever since we started using cucumber, i have never seen its practical benefit for our community. And i already emphasised this from the very beginning of our cucumber integration:
I like this. But in order to start simple, i would deal with it after all the others are done.
The overhead and unnecessary dependency of cucumber will ultimately pause a risk to slow down the CI/CD and qa framework process. Most of the qa engineers have actually been trying to overide the aim and goal of Cucumber creator, Aslak Hellesoy. He said in one post;
If you think Cucumber is a testing tool, please read on, because you are wrong.
Cucumber was born out of the frustration with ambiguous requirements and misunderstandings between the people who order the software and those who deliver it.
openmrs we need to abrace ourselves with automation best practices.
However, it took 38 mins to run the GitHub action (16 mins to spin up the container, 17 mins to run tests, and 5 mins for other tasks)
We can decrease the time by either
run it on bamboo (not sure how to run a bamboo plan with a PR)
or configure tests to run without a container in GitHub.
run on bamboo, run tests parallelly
run on GitHub without a container, run tests parallelly
My rough time estimation for each above options are
17 min (the container spins immediately )
17 min (no need for a container)
4 min (tests runs parallelly; est. time = max(time taken by a spec))
4 min (same as above)
Currently I’m trying to evaluate Cypress and Playwright. We are using both tools right now; playwright for offline tests and cypress for other tests. Both tools provide the same functionality, but Cypress has some issues with offline testing. Tests were already implemented with cypress when the offline testing requirement came. That’s the reason for having two tools. Btw, I found a nice comparison.
As a first step towards migrating tests to relevant repositories, the following PR aims to ensure that tests are located in the same repository as the code they are testing and can be run automatically as part of pull requests or individual commits. This will provide a more streamlined and efficient testing process, allowing tests to be run automatically as part of the development workflow. The PR addresses the requirements outlined in the associated Jira issue.
