Version-Aware E2E Testing: Parameterized Docker Image Tags in OpenMRS Test Infrastructure

Development Quality Assurance
1

Hey team! :waving_hand:

I’ve been looking at our 03 E2E testing setup, and I think we have a great opportunity to make it more flexible and version-aware. Right now, our tests always run against the latest nightly builds of OpenMRS Docker images (openmrs/openmrs-reference-application-3-*:nightly). While this ensures we test against recent changes, it doesn’t account for version compatibility—meaning we can’t easily:

  1. Reproduce bugs reported in older OpenMRS versions (e.g., “This broke in 3.2.1”).
  2. Verify backward compatibility before releases (e.g., “Does our new feature work in 3.1.0?”).
  3. Test against stable releases instead of relying solely on nightly, which might include unreviewed changes.

I’ve prototyped changes to:

  • Parameterize Docker image tags (e.g., :nightly → :${TAG:-nightly}) so we can override versions.
  • Modify the E2E script to accept version args (e.g., APP_SHELL_VERSION=3.2.1).
  • Update GitHub Actions to support manual triggers with version inputs

Benefits:

:white_check_mark: Test against specific versions (e.g., 3.1.0, 3.2.1) on demand.

:white_check_mark: Keep the default nightly behavior for CI.

:white_check_mark: No disruption to existing workflows.

Thoughts from the community:

  1. Does this align with our testing goals?
  2. Should we add scheduled jobs to test against stable versions automatically?
  3. Are there other components (e.g., APIs, modules) we should version-lock in tests?

Example of triggering a version-specific test:

APP_SHELL_VERSION=3.2.1 BACKEND_TAG=3.2.1-with-data ./run-e2e-docker-env.sh

Any thoughts?,

cc @ibacher @jayasanka @dkigen @dkayiwa

3 Likes
2

Basically, all of this can be done sans having the versioned -with-data images.

I don’t think backwards compatibility of the sort you’ve described is something we’re currently aiming to support.

I also don’t think that GitHub Actions is the right place to be running these. We can do them on Bamboo or locally.

1 Like
3

Thanks @ibacher for the feedback!

That makes sense—appreciate the clarification on current priorities. I’ll adjust the setup to focus on keeping the version parameterization available for local and Bamboo runs only, and leave GitHub Actions as-is for now. My rationale for GitHub Actions was:

  • Speed: Parallel testing across versions (if we ever need it).
  • Dev UX: Manual triggers with dropdowns for versions

Happy to align with that. Will draft a documentation for the version override option for local debugging or Bamboo use.

1 Like