@k.joseph@christine, based on the QA Team Call today, @dkayiwa and I would like to propose the following. Can you let us know what you think?
Goal: Increase automated RefApp test coverage quickly to avoid further valuable dev time being used on repeated manual UI tests (like we did in this list last sprint - that was helpful but won’t last long before we have to test again), and to save ourselves time in the near future. There are a lot of tasks still to do for the RefApp release, and we definitely don’t want to have to repeat manual tests after each code commit.
Proposed steps to achieve this:
Automation Documentation (Urgent): A simple, step-by-step guide on how to write a selenium test, that shows and example and a pattern to follow. @k.joseph could we have this done this week so people could start using this guidance asap?
Design Session: Once the documentation/mini-guide is ready, we can use a weekly design session as a practice time for community devs to join and practice writing selenium tests.
With big frontend changes in our future, it will serve us very well to have lots of community knowledge about how to create and maintain automated UI tests.
Interesting feedback from @bistenes: He recommends we use Cypress and not Selenium—Cypress can do everything Selenium can do for old fashioned apps (like RefApp 2.x). And it’s what we’re using for Frontend 3.0 / MFE. So people can get started learning Cypress now, instead of ever having to learn Selenium, which is legacy technology at this point. @dkayiwa would you be ok with proceeding with Cypress instead? I think all our OCL for OpenMRS tests were built in Cypress as well, but then we ran into challenges because not many people were familiar with Cypress. Sounds like a good idea to move forward with building test automation knowledge and Cypress familiarity at the same time.
.
Good catch from @bistenes@grace i think qa lead @k.joseph@christine need to discuss about these two frameworks because seems they have added some automation test in selenium effectively otherwise thanks alot
This is a very Awesome idea. Selenium is really simpler for new Devs but with enough information on how to write tests in cypress can we can still move on with it if its better.
one issue with Selenium is the structure of feature, page and steps files which are interdependent. when these are written by different people it really slows down the process of writing a test case for one feature and I think it has affected us too of recent . i dont know if its the case with cypress too
Cypress is great and lighter-weight than Selenium, but I’ve got a bit of hesitation around whether it’s the right choice for something like the RefApp 2.11 release. Basically, my concern boils down to this: we have some community experience with Selenium, but only a small amount with Cypress. Further, we can drive Selenium with Java code, and there’s a much greater wealth of Java programmers in the community than there are Javascript programmers (Cypress, quite rightly, only has a Javascript API). In other words, I’m wondering if we have the capacity to create effective Cypress-based tests. If we have someone with the right skills and availability to setup a clean Cypress-based framework, I’m all for it.
Thanks @bistenes for the insights, what do you think of Selenium’s WebDriver protocol which enables sending commands in various development languages — like Java, Java Script, C# than Cypress which uses java script.
sure @grace the design sessions would greatly help dev to get up to speed with selenium test writing .
In the mean time as the documentation on this is being generated the community devs ready to participate can be getting them selves ,more familiar with selenium and its tests, there is a whole lot of resources out there on this, so that the design sessions will be a quick through and people will easily get down to business
The QA team had dialogs in earlier stages and compared key frameworks such as selenium, cypress and puppeteer and even did POCs for each integrated with cucumber and from that effort and dialogs selenium was preferred.
IPV, the language in which the ui tests are written isn’t a big deal except that OpenMRS has more backend developers but perhaps frontend-3.0 folks are opting for a front end based framework because the projects is so.
However to me cypress has more cons than selenium such as one browser, language category, test runner support and maturity stage in comparison to selenium.
About the performance speed, most articles call cypress faster than selenium but except if the ocl setup wasn’t great since the 14 selenium tests at https://ci.openmrs.org/browse/CONTRIB-QA-230/log run for 1 min 31.99 secs while the 11 cypress ocl tests at https://travis-ci.org/github/openmrs/openmrs-ocl-client/builds/727089491 run for 6 mins 46secs.
This dialog can be resurrected perhaps.
i think it is wise for front end squads such as OCL & micro-frontends to use something like cypress since the contributors are more familiar with that, our long time contributors and investment however is still an urgent to keep supporting selenium (java based)
Yes, for any modern frontend project, Cypress is the obvious choice, and Selenium would be a mistake.
For old-fashioned websites (i.e. websites in which the frontend is rendered by and inseparable from the server), such as the RefApp, the debate makes sense. Selenium would be the standard choice.
Developers should be aware that Cypress is the future of frontend integration testing. Learning it it is a matter of keeping up with the industry. Fortunately, Cypress code is much more brief, clear, and easy to read than Selenium code, since it has a clean, modern API. The transition is a pleasant one.
Nevertheless, I’ve never tried using Cypress on an old-fashioned website, so I can’t vouch for it in that kind of context. I expect that it would work better than Selenium (which is prone to intermittent and inexplicable failures), but I wouldn’t stake anything on it.
IMO the bottom line is that it’s fine for the QA team to use the good old Selenium with Ref App 2.x, as long as it’s on their roadmap to add Cypress to their toolkit when Ref App 3.x falls in their scope of responsibility.
@bistenes and @mksd, have you tried evaluating puppeteer against cypress? am wondering if there won’t be any thing upcoming to make cypress an ex tool considering how fast the front end has changed in the past 5 years.
I haven’t! I don’t think E2E testing with Puppeteer is very common in the frontend world at this point. Probably because Puppeteer is a lower-level tool, which together with an assertion framework (like Jest) and a mocking framework could constitute a good testing solution.
If you zoom out in that graph to “5 years” you’ll see that they originated about the same time and both have consistent growth, though Cypress’s is steeper of late. If Cypress is going to be obviated by anything, it’s really not clear yet what it would be.
We’ll of course reevaluate when it’s time to start QAing OMRS 3.0. But I do not think we will regret ditching Selenium.