Previously, we have been using Google Sheets to manage our test plans and test executions for both Refapp 2.x and 3.x. However, after some research, we discovered a Jira Plugin called Xray that simplifies the process of test management and automation. Additionally, Xray offers a Bamboo integration, which allows us to optimize the CI with automatic test report generation.
By utilizing Jira Xray, we can effortlessly create, organize, and execute our manual tests. This enables us to conduct tests as and when required, such as before a major release or once every month. I have created a brief demo to demonstrate how we can use Xray to manage manual tests in OpenMRS.
Furthermore, I have also created a document that provides an overview of the basics of test plan execution.
I have also established a demo Jira instance at https://test-omrs-qa.atlassian.net/ for demonstration purposes. Please let me know if you need access to test it yourself. Feel free to share any thoughts, suggestions or questions you might have.
I would also like to mention that we currently have a similar tool called Zephyr installed in the OpenMRS Jira instance. Hence, I will be comparing the pros and cons of both Xray and Zephyr to identify the most appropriate tool.
Thanks for the observation @piumal1999 ! Exiting to hear about the Zephyr comparison. Meanwhile, I’ll look into the SSO migration, which blocks us from upgrading the JIRA version.
Hello all, I have discovered that the current Zephyr version installed on the OpenMRS Jira instance is Zephyr for JIRA 5.6.4, which is a discontinued edition of Zephyr. Moreover, Zephyr no longer provides support for that version.
Therefore, currently, there are two editions of Zephyr available:
Zephyr Squad: This test management solution works within Jira and allows teams to plan, execute, and report on their tests.
Zephyr Scale: This standalone test management solution helps teams create, manage, and automate their testing process.
I have briefly reviewed the documentation of both solutions and also tried installing them on a Jira instance. The main difference between the two is that Zephyr Squad enables tests and executions to work as Jira tickets, while Zephyr Scale is an isolated plugin that doesn’t directly work with Jira tickets.
As for my comparison of Xray and Zephyr, here are my findings:
Similarities between Zephyr and Xray:
Both offer basic features such as creating, editing, and cloning tests.
Both have a visual step editor.
Both natively support Gherkin.
Both have hierarchical test organization.
Both integrate with Jenkins and Bamboo.
Both offer licenses for open-source organizations.
Both have test plans (Zephyr Squad doesn’t have this).
Advantages of Xray over Zephyr:
Xray can use test sets for organizing, whereas Zephyr does not have this feature.
Xray uses different Jira ticket types (Pre-Condition, Test, Test Set, Test Execution, Test Plan) for managing tests, while Zephyr only uses the Test ticket type.
Xray has test plans, which Zephyr Squad does not have.
With Xray, we can report issues and bugs in a very detailed manner while executing tests.
Advantages of Zephyr over Xray:
The test step UI is simpler and easier to use than in Xray. The Xray UI is more advanced since it has many features.
Based on the comparison between Xray and Zephyr, it seems that Xray has several advantages over Zephyr, including the ability to use test sets for organizing, the use of different Jira ticket types for managing tests, the presence of test plans, and the detailed issue and bug reporting. Therefore, my recommendation would be to use Xray for test management in OpenMRS Jira.
I recently discovered that we can use Xray to display the results of our automated Playwright tests. By importing the generated JUnit report, Xray automatically creates a test execution report (dashboard) with the results and adds any new tests to the relevant test plan. To set this up, all we need to do is configure the e2e test GitHub action to send the results to Xray via the Xray API. You can find a detailed tutorial on how to do this here:
Furthermore, we can create a “trigger test” button on the Xray test plan. This button allows us to initiate tests directly from Xray and triggers the e2e test GitHub workflow, making it easier to manage and execute tests. You can find the tutorial here. There’s a similar integration for bamboo as well.
By the way, we may need to identify the specific use cases in OpenMRS development that require these features.
Ok. We try to limit Atlassian plugins to the bare minimum, since we have limited people/time to maintain our infrastructure and plugins can make things less stable.
Given that Zephyr isn’t being used, switching to Xray is reasonable. I’ll go ahead and uninstall Zephyr, install Xray, and report back here.
Well, bad news. I’ve uninstalled Zephyr from JIRA and tried installing Xray; however, Xray does not support install on Server (self-hosted) versions of JIRA. We won’t be able to use Xray until we migrate to Atlassian cloud.
There aren’t any current plans for migration. We first need to work out a solution to make OpenMRS IDs to work with Atlassian Online.
We’re investigating SSO with Atlassian Online using a tool like KeyCloak, but need to work out a solution (e.g., postfix) to map email usernames from a custom domain (e.g., user.openmrs.org) via corresponding LDAP username to email in LDAP.
Meanwhile, I’ll look into the SSO migration, which blocks us from upgrading the JIRA version.
