Cloud Vs Private Web Testing Platform?

Development Quality Assurance
1

There are a number of good cloud based web & mobile test platforms such as; BrowserStack, saucelabs and the likes, these are great in allowing cross browser, version and operating system testing and offer a lot more spices for automated end to end testing.

From my historical sketches, we have saucelabs integrated in our reference application ui tests, i wonder whether we hold a free license or what @raff since you own these! From my analysis the main challenge about these is the fact that the best are commercial.

Am trying to analyse those cloud based vs our own community built and maintained platform.

We could go around the commercial challenge with some good virtualisation platforms such as docker, the choice of a platform to own and spin our test cases affects how we write our testing framework though, Selenium Grid architecture for-example allows us the benefit or testing against multiple browsers, browser versions and operating systems. once Selenium is favoured i would appreciate reusing the reference application ui tests, the extra effort would be wrapping selenium Grid in docker rather than routing tests through saucelabs and then writing extra test cases for identified user requirements. We can still hook that up into CI suchas Jenkins CI, Bamboo to trigger running of and managing test case results there!

Please drop your thoughts around these 2 options else help shape a better approach.

2 Likes
2

so glad learning about saucelabs providing their full-service to open source organisations for free. @raff, did you request a free account through this provision?

Perhaps supporting both packagings makes the framework able to be triggered when changes are detected and also make it easier for implementations to test with the same locally!

1 Like
3

@k.joseph Overall I would recommend avoiding to roll out infrastructure, when there are free (or even paid) options as this saves expensive “human” time to maintain and evolve.

I for one would rather have greater test coverage for more confidence in the platform & ref app than control over the infrastructure

1 Like
4

I also found https://github.com/openmrs/openmrs-contrib-uitestframework That is 2 years since the last commit, so I guess it is not up-to-date or used “for real” in automated tests.

5

And the old thread Sauce Labs for QA Team & testing? has various discussion from the time that SauceLabs use seems to have started. Linking here because it might be useful to read…

6

Saucelabs gives 5 concurrent sessions free to Open Source projects. Travis (for example) also runs 5 jobs in parallel for free for Open Source projects. That works in nicely, because you can make however many test-suites you like (for lots of UI features/scenarios) and run each suite in a separate Travis job. 5 suites will run at a time in Travis, creating 5 sessions in SauceLabs. So you don’t get “session limit exceeded” problems in SauceLabs.

If the CI gets linked from PRs in development, a PR creation/push of commit/… could potentially run a job that builds the resulting code, generates a whole new reference app environment, then runs UI/API/end-to-end tests against that new reference app in Travis+SauceLabs. The PR can get a pass/fail result

    • substitute CircleCI or your other favourite free-for-open-source online CI tool for Travis…

Is CI hooked up that much yet?

1 Like
7

I’d echo @ssmusoke’s comments here, especially as there are at least some automated frameworks that are free for open source projects.

I’d also like to toss my vote in for Selenium over Puppetteer. Puppetteer looks exciting, but Selenium is clearly the more mature and feature complete product and closer to an established industry standard. More importantly, I think Selenium stands a better chance of attracting volunteers with existing web testing knowledge, since they will likely have some experience with it and is better documented through tutorials, etc. which could help getting new testers on board, which contributes to the long-term sustainability of an automated testing program.

1 Like
8

I think the more important piece is whether we can hook this into our Bamboo builds that are used for releases and CI for key modules. My sense is that OpenMRS uses Travis, etc. (where they are used at all) for quick non-integrated tests of a single module.

9

the beauty is the current reference application ui testing framework (CC @raff ) is run by both travis and saucelabs through the same CI on every commit

1 Like
10

I guess that the reference application repo has the “custom” reference application “settings/forms…” plus a “latest” copy of the core OpenMRS code (either committed in the repo, or sucked in/built on-the-fly in CI). When there is a PR to the reference application repo, then I see that CI runs, which runs the UI tests - good stuff.

What happens for PRs to “core” code repos? Do those manage to trigger a “build” of the reference application and a run of the UI tests?

How does a developer know, at the time they submit a “core” code change, if it might break any reference application tests?

11

Just want to second (or third) this…