Dear QA Team,
Several years back when we first looked at functional testing, we tried Sauce Labs to run Selenium tests in their cloud with different virtual browser combinations. It was nice, but we weren’t really ready for it back then.
They have quite a few useful features:
It’s free for open source projects. Should we take a look at using it again? If so, let me know, and I can coordinate getting it set up.
Given issues we keep running into executing UI tests by our CI I say “YES, LET’S TRY SAUCELABS”.
I’ve sent sub-account invitations to @ryan & @raff … if anyone else wants one reply to this topic.
This tutorial explains how to use integrate Atlassian Bamboo with tests run with the Sauce Labs cloud of Selenium servers: https://docs.saucelabs.com/ci-integrations/bamboo/
That looks quite promising. It would be awesome to run Selenium tests on lots of combinations of browsers and OS’es.
I wish I had time to help out, but I’m still interested. What do you mean by “sub-account invitations”?
We have a general account for Sauce Labs for OpenMRS that’s set up for the free open source project access level of the service. I can send invitations for individuals to sign up for “sub-accounts” to access those privileges and the tests that we set up as a group.
I’m ok with it if it gives us more information and stability than CI. I would also like to sign up for a subaccount.
I have successfully run UI tests on saucelabs!
Our CI server runs 69 tests in 29 minutes. It took 45 minutes to run 69 tests on saucelabs, however we could try to run on saucelabs 5 tests in parallel, which would shorten the time to 10 minutes or so. Tests are run on separate VMs, which are discarded after each test. It is why it takes a bit longer to run them sequentially.
The service has a nice integration with Bamboo, see https://docs.saucelabs.com/ci-integrations/bamboo/ It provides access to screencasts from test executions. It gives the possibility of running tests against different browsers and OSes with no extra work.
One drawback is that locally tests would have to be executed using the old way since running them on saucelabs requires authentication, thus a subaccount is needed. As I understand, it can be handed off to a limited number of people.
I would say that if we can get saucelabs to run our tests in parallel and shorten the time to 10 minutes then let’s use it.
Here are the changes I needed to make to run our tests on saucelabs: https://github.com/openmrs/openmrs-contrib-uitestframework/compare/saucelabs (authentication details in TestBase are fake and must be replaced before running)
@natalia, if you want to try running tests on saucelabs locally yourself, then yes, pull my changes and set your login and token in TestBase. However, the changes are not yet ready to be merged into master. First we need to find a way to adjust the configuration so that it is possible to run UI tests without an account on saucelabs and also to run tests on saucelabs in parallel. I don’t have time to look into that right now myself, but you are welcome to try.
I assume when we’re ready, we would eventually create a separate account & access key for the tests themselves rather than have them tied to an individual user?
Hi, how the implementing Sauce Labs on CI is going?
@tmueller, I haven’t had a chance to look into that. It’s not that much about implementing it on CI rather taking the changes , which I did above a step further by introducing a way to run tests either on Saucelabs or locally. We could switch between the two methods using a maven profile. It would be great to configure tests in a way that when they run on Saucelabs they execute in parallel and make use of 5 servers saucelabs provides us.
You are very welcome to work on that as I’m focused on other tasks right now.
@raff, I tried running tests locally using Sauce Labs implementing your changes: https://github.com/openmrs/openmrs-contrib-uitestframework/compare/saucelabs3 and on sauce labs page when seeing test results I am having an error: “Test did not see a new command for 90 seconds. Timing out” for every test. Did you face any issue like that?
I may have overlooked that, because maven said all tests passed. I think the issue might be that we use the same webdriver for all tests in a class. I don’t think it is correct even for running tests locally.
You should try changing @BeforeClass to @Before and @AfterClass to @After in https://github.com/openmrs/openmrs-contrib-uitestframework/blob/master/src/main/java/org/openmrs/uitestframework/test/TestBase.java Pay attention to the execution order of @Before and @After methods. See http://stackoverflow.com/questions/6076599/what-order-are-the-junit-before-after-called startWebDriver must be executed always as first and stopWebDriver always as last.
Ok, I managed to run tests on Sauce Labs locally. Now I’m working on running them in parallel
I’m trying to follow this example: https://github.com/ndmanvar/Java-Junit-Selenium/blob/master/src/test/java/com/yourcompany/SampleSauceTest.java
annotating but I keep getting this error:
java.lang.IllegalArgumentException: wrong number of arguments
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.saucelabs.junit.ConcurrentParameterized$SauceClassRunnerForParameters.createTestUsingConstructorInjection(ConcurrentParameterized.java:195)
at com.saucelabs.junit.ConcurrentParameterized$SauceClassRunnerForParameters.createTest(ConcurrentParameterized.java:190)
at org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:244)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:241)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at com.saucelabs.junit.ConcurrentParameterized$SauceClassRunnerForParameters.access$300(ConcurrentParameterized.java:170)
at com.saucelabs.junit.ConcurrentParameterized$SauceClassRunnerForParameters$2.run(ConcurrentParameterized.java:287)
at com.saucelabs.junit.ConcurrentParameterized$NonBlockingAsynchronousRunner$1.call(ConcurrentParameterized.java:312)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
At first glance it seems it expects the test to have a constructor with arguments matching what the browsersStrings method returns, i.e., public SampleSauceTest(String os, String version, String browser, String deviceName, String deviceOrientation)