Evaluation of our current QA Pipeline

Over the last week, I attempted to evaluate our automation technical pipeline, debt, and opportunities. There, I focused on E2E tests because we already have mechanisms in place for unit and integration testing, but E2E tests aren’t well-established yet. Another reason is that, when considering the openMRS architecture, E2E tests provide more accurate feedback than unit and integration tests because they run in a full real-life environment rather than a mock environment.

I reviewed our current approach as well as the documentation of other product teams on E2E test automation, such as Hackney, Wix, and Single SPA, to determine what might be useful for the OMRS application. There, I discovered the following flaws in the current workflow.

  • E2E tests are not a part of the devs’ workflow.

    The developer engagement is extremely low with the current setup. One reason would be that we aren’t providing any immediate feedback for developers. Therefore developers don’t have an idea even if their changes break tests; i.e. break the product. There are countless cases where we have identified critical breakdowns in components weeks after PRs have been merged.

  • Unclear Guidance

    Best practices, and our expectations around testing, are not clearly documented.

  • Cucumber is slowing down O3 test automation

    What Gherkin syntax is really good for is if we have a less technical team to implement tests. You can have a team of developers create a library of steps and then have less technical users write the scenarios, which is not how we work. Everything we’re getting with Cucumber can be done with a good naming convention for tests and having tests properly organised into suites. Making sure that non-technical users can read a report of tests and understand what parts failed; is a crucial component of effective test design. In O3 we use a separate library to integrate cucumber with cypress to have a bridge we can maintain. In order to achieve that, we are sacrificing some cypress fx, especially as cypress upgrades. Those libraries are not well maintained.

Yesterday, @ibacher , @dennis , @jnsereko and I had a call and discussed the above further and identified the following as the next steps:

  • Fix failing tests and enable tests in the deployment pipeline

    We already have a workflow to run E2E tests on deployment. In order to get a better result, first we need to make sure all the tests are up-to-date with the current implementation.

  • Improve DevX

    In order to make the E2E framework sustainable we need to improve the developer experience.

    • Evaluate the current tools and technologies we use for E2E testing.
    • Provide immediate feedback to developers
    • Store tests in a convenient place The farther away the tests from the codebase are, the fewer attention developers pay to it. We need to make sure tests are placed in a place where developers can easily update the test with their changes. One possible solution would be to have tests in each micro frontend repository itself. So that developers can easily update tests along with their PRs. It is how wix.com address this issue.

Thanks for reading, let me know your feedback! :hugs:

cc: @dkayiwa @grace @jennifer @christine

2 Likes

@jayasanka, this is a very useful evaluation!

FWIW, with openmrs-core 2.6.0, we have introduced Testcontainers such that one is able to run all existing tests on MySQL with: mvn test -DuseInMemoryDatabase=false -Ddatabase=mysql or on PostgreSQL with: mvn test -DuseInMemoryDatabase=false -Ddatabase=postgres

As part of the next steps, do you also plan to improve the documentation?

This is one of the best paragraphs that you have ever written for my consumption!!! Ever since we started using cucumber, i have never seen its practical benefit for our community. And i already emphasised this from the very beginning of our cucumber integration:

I like this. But in order to start simple, i would deal with it after all the others are done. :slight_smile:

2 Likes

created an issue for this [O3-1622] Remove Cucumber from the RefApp 3.x Testing tooling - OpenMRS Issues

1 Like

The overhead and unnecessary dependency of cucumber will ultimately pause a risk to slow down the CI/CD and qa framework process. Most of the qa engineers have actually been trying to overide the aim and goal of Cucumber creator, Aslak Hellesoy. He said in one post;

If you think Cucumber is a testing tool, please read on, because you are wrong.

Cucumber was born out of the frustration with ambiguous requirements and misunderstandings between the people who order the software and those who deliver it.

openmrs we need to abrace ourselves with automation best practices.

Thanks for all of your responses! :heart:

Yes that would be a priority too.

Here is a quick update on the current state:

  1. Tests are fixed. Distribution 3.x E2E and scheduled tests on GitHub are passing now. The next step is to enable the pipeline to run on commits.

  2. I was able to run E2E tests against a PR. Link

    However, it took 38 mins to run the GitHub action (16 mins to spin up the container, 17 mins to run tests, and 5 mins for other tasks) We can decrease the time by either

    1. run it on bamboo (not sure how to run a bamboo plan with a PR)
    2. or configure tests to run without a container in GitHub.
    3. run on bamboo, run tests parallelly
    4. run on GitHub without a container, run tests parallelly

    My rough time estimation for each above options are

    1. 17 min (the container spins immediately )
    2. 17 min (no need for a container)
    3. 4 min (tests runs parallelly; est. time = max(time taken by a spec))
    4. 4 min (same as above)
  3. Currently I’m trying to evaluate Cypress and Playwright. We are using both tools right now; playwright for offline tests and cypress for other tests. Both tools provide the same functionality, but Cypress has some issues with offline testing. Tests were already implemented with cypress when the offline testing requirement came. That’s the reason for having two tools. Btw, I found a nice comparison.

    Cypress vs Playwright: The Rematch - YouTube

1 Like

Thanks for your work on this!

Do we create DB from scratch? If we import a DB dump instead, we should significantly reduce the time needed to startup.

1 Like

Yeah, it’s creating the DB from scratch. Maybe that’s what slowing down the startup. I’ll give it a try. :hugs:

1 Like