I’d like us to implement automated performance testing for Reference Application. Tests would be run as part of the validate stage at https://ci.openmrs.org/browse/REFAPP-OMODDISTRO/latest
@cintiadr, would it be possible to create a dedicated bamboo agent, which could be used exclusively by a new performance test task? The agent would need to run on a dedicated machine without any other heavy processes so we get reliable results.
Should be possible, I believe. I’d ask @burke about allocation in Jetstream, but from my point of view it should be alright.
I’d argue this machine could potentially run other builds too (unless you have reason to be strongly against it). While one a single build should run on that machine, while it’s idle I don’t see why other builds shouldn’t run there.
But reality is that, while the work is not so hard (it would be go to terraform, setup a new machine that now bootstraps ansible AND puppet), I will not have time to do that any time soon.
On the other hand, I’m happy to help someone willing to go that path.
On the other hand, if you just wanted a dedicated openmrs server (let’s call it perf-ref.openmrs.org) in docker, it would be pretty trivial.
But then the question would be if docker is a good medium to test the application.
hi @raff, great initiative! Could you share a bit on what tools are you considering (maybe http://gatling.io/ ?) and what things you would like to measure?
@cintiadr, I guess it’s okay to run other builds while the agent is idle. I’m not sure how Jetstream allocation works, but the main point here is to set it up in a way, which guarantees consecutive builds of the same commit will run in same conditions (server and tests will be run on the same agent to ignore network) and tests complete in similar time with ~5% difference at most.
I guess we could start from running them on existing agents and see if they are reliable enough. I’ll reach out to you for guidance regarding setting up a dedicated agent if needed.
@teleivo, I wanted to focus on testing REST endpoints so that we can cover platform, RA and any other distribution with a single test suite, which could be then extended. I haven’t decided on which framework to use. I’ll be looking at https://www.joecolantonio.com/2017/07/18/open-source-performance-testing-tools/ to start with. Gatling is on that list. I appreciate any suggestions.
Originally I thought about using https://github.com/rest-assured/rest-assured with JUnit and http://openjdk.java.net/projects/code-tools/jmh/ or http://jmeter.apache.org/usermanual/junitsampler_tutorial.html for simplicity, but we can go with something more robust.
One important feature for us is the ability to send out warnings for particular commits, which introduce inefficiencies.