Test Coverage for our REST API

@kdaud do not put tickets in the post commit state because the pull requests are not yet merged. You may need to take a second look at Pull Request Tips - Documentation - OpenMRS Wiki

@dkayiwa I was not careful while moving the tickets from In Progress to Ready For Review.

I think that tells me what I was supposed to do while changing the tickets state. I am now a consultant and have already become smart in that angle :sweat_smile:

Here is the research on the tools above.

All the tools mentioned are open source frameworks though they differ in the features they offer in respect to API. Below is the summary of what each framework offers!

Swagger is a tool which supports a functional, performance, and security testing of API.

Swagger Features:

  • API design and development
  • API documentation
  • API Testing
  • API Mocking and Virtualizations
  • API Governance and monitoring

Best For: API designing.

Karate is based on the cucumber library and supports a web service test written in a domain-specific language(DSL).

Karate Features:

  • Multi-threaded parallel execution is supported.
  • Allows configuration switching.
  • Reports generation.
  • Supports reusing Payload-data for API testing.

Best For: Supports any language which can deal with HTTP, JSON, or XML

Jmeter supports cross-platform and works at a protocol layer.

Jmeter Features:

  • Allows use of various programming languages.
  • Load and performance testing of many different applications, servers, and protocols.
  • Allows to replay test results.
  • Provides support for variable parameterization and assertions.
  • Supports per-thread cookies.
  • Configuration variables and a variety of reports are also supported by Jmeter.

Best For: load and performance testing of web applications.

Postman is an API development environment however, its not free. It has three pricing plans which I won’t go into details for now.

Postman Features:

  • Helps in automated testing.
  • Assists in exploratory testing.
  • Supports Swagger and RAML (RESTful API Modeling Language) formats.
  • Supports knowledge sharing within the team

Best For: The tool is best for API testing

After looking into the existing tests in openmrs-module-webservice.rest, I noticed they are using REST-Assured framework and below are some of its features

  • Seamless integration with the Serenity automation framework.
  • Provides some baked-in functionalities.
  • Supports the syntax of BDD Given/When/Then.

Thoughts are welcome on this as this is guiding our move into automated tests for REST API direction.

cc: @sharif @dkayiwa @grace @ibacher @bistenes @christine

1 Like

So, do you have a conclusion on what would be the best way forward?

From today’s QA call, two of the framework have been selected(Karate and REST-Assured) however, one of them will be opted for after more research is done finding out which of the two is better in terms of sustainability and can support multiple languages(basically Java and Js).

Hopefully the decision will be taken in the next QA Call.

We may harness this as well

Sustainability is an important goal. I’m less convinced that support for multiple languages is something we need if we’re talking about the language we write the tests in. (This is separate from which language the application itself is written in; for ex. Selenium tests can be written in Java but drive a Javascript-based application… or a Python-based one, etc.).

One thing I didn’t see much of in your review above was on the test reporting from these tools, i.e., can they generate results in some kind of standard format that can be easily used by other tooling to indicate how many tests passed or failed and which tests? (Most helpful would probably be something like the JUnit format).


thanks, @tendomart this is really helpful

@sharif @tendomart as we investigate on the tools that were selected, worth to consider @ibacher’s point above !

Yes @ibacher @kdaud , Karate provides an option for reporting format, it provides an integration with cucumber reporting plugin which helps in formatted generated report, something like

`        ` `<groupId>net.masterthought</groupId>`
`        ` `<artifactId>cucumber-reporting</artifactId>`
`        ` `<version>` `3.8` `.` `0` `</version>`
`        ` `<scope>test</scope>`

We can generate reports in HTML format, json format ,XML format as well, though the configuration especially in test runner class seems to be little deeply than selenium

The way forward on the REST API automation is follows:

  • REST-Assured will be used to complete the automation of the pending REST API tests. The team agreed there was no need to change the framework as we already have a large number of the API tests automated using the framework. Additionally, the tool is working well for the community. @kdaud has already completed 7 test and we are pending 6 tests to complete the automation.

  • The review of the other suggested tools will be completed later on and the information kept in file for future reference or when a need arises .

1 Like

All the REST API tests that were identified missing by the Platform team have been automated and now running on every PR triggered on the repo and on every merge. We added more few tests that were identified during the development. Checkout the updated Spreadsheet to see these automated tests marked in green as Done(second column) and the links to their implementation are provided in the first column. All the tests status are tracked on our QA-Dashboard under the REST API budge which tracks the current Bamboo build of the module.

Thanks to the QA Team that has worked around ensuring the project succeeds!

cc: @dkayiwa @tendomart @christine @grace @jennifer @sharif