Shifting to using the correct environments for testing

We are (incorrectly) using devtest01 and devtest02 for automated and manual testing of the reference application.

Rather than being devtest these should be qa or uat or something.

Michael Downey brings this up from time to time and I agree with him every time, but I don’t have time to work through all the details. Does anyone out there have done spare cycles to take this on?

I presume the steps involved are:

  1. Decide, with Michael, what the servers should be called, and get them set up by help desk.
  2. Make changes to CI to point to the new servers
  3. Update wiki documentation
  4. Turn off the old servers, see what breaks that I have missed here.

Anyone?

3 Likes

Thanks in advance to people thinking through this! :+1:

Here’s a handy reference chart of “what’s what”:

  1. Development Test environments: Development servers, also known as “sandboxes
  2. Integration Test environments: CI build targets, or for developer testing of side effects
  3. Quality Assurance Test environments: For functional, performance testing, Quality Assurance etc.
  4. User Acceptance Test environments: For user acceptance testing purposes

Something like this?

  • int01 – OpenMRS Platform built from latest commit to core or any modules bundled in the platform. Purpose is to prove OpenMRS Platform still runs and to serve for automated testing.

  • int02 – OpenMRS Reference Application built from latest commit to core or any module distributed in the ref app. Purpose is to prove OpenMRS Community Distribution still runs and for automated testing.

  • int03+ – Used whenever we need an additional server for automated testing (e.g., automated testing of a new module that doesn’t belong on or might unnecessarily disrupt the operation of int01 or int02).

  • qa01 – OpenMRS Platform built from latest commit. Used for manual testing (e.g., making manual REST calls to it). This would be similar to a headless version of demo.openmrs.org, but would be running a version built from the latest commit (to core or any module bundled in the platform.

  • qa02 – OpenMRS Reference Application built from latest commit. Used for manual testing. This would be similar to demo.openmrs.org, but would be running a version built from the latest commit (to core or any module distributed in the ref app).

  • qa03+ – Used whenever we need to perform manual testing for an application or module that doesn’t belong on or might unnecessarily disrupt the operation of qa01.

  • uat01 – OpenMRS Platform from manually triggered build. Purpose is to run a specific snapshot of the platform released to uat01, so specific releases could be user-tested during preparation for a new release.

  • uat02 – OpenMRS Reference Application from manually triggered build. Purpose is to run a specific snapshot of the reference application released to uat02, so specific releases could be user-tested during preparation for a new release.

  • uat03+ – Used whenever we are releasing something other than the reference application and need a server onto which trigger new builds as we work through user acceptance testing of the impending release.

If we agree, then:

  • CI should be deploying “nightly” (build from latest commit) of platform to int01 & qa01 and the reference app to int02 and qa02.

  • Automated testing by CI of platform or reference application should be performed against int01 and int02, respectively.

  • Any /dev/3+ should be able to manually trigger a new release (of a particular commit) of the platform to uat01 or the reference application to uat02.

I think we should also have qa and uat environments for the OpenMRS Platform release. That needs to be tested also. :slight_smile:

Further, is there some reason we can’t give the servers more meaningful names in this scenario? E.g. int-platform and int-refapp instead of int01 and int02.

Noted and my list updated accordingly.

Agreed. Alternatively, we could use aliases (e.g., int-platform.openmrs.orgint01.openmrs.org).

It would nice to stay a little consistent with the numbering where the ones that have a 1 e.g int01, qa01 etc are for the platform deployments while those with the 2 e.g int02, qa02 are for ref app deployments

Ideally somewhere we’ll get to the point that we do enough testing that we’ll need multiple environments per project. Particularly for UAT.

Ping.

I expect that the infrastructure team will own making these changes. Is this waiting on anything besides getting prioritized?

I can probably do part of that, but I’m still completely lost.

So, this is the current status.

Distribution build - deploys to devtest01.openmrs.org and after the tests, devtest02.openmrs.org (automatic) Distribution/Deploy OpenMRS 2.x/DTE 03 and Ebola Example module/Deploy Ebola - deploy to devtest03.openmrs.org (manual) Distribution/Deploy OpenMRS 2.x/DTE 04 - deploys to devtest04.openmrs.org (manual)

Core Master/Nightly build server - deploys to int01.openmrs.org (automatic)

Distribution/Deploy OpenMRS 2.x/UAT01 - deploys to uat01.openmrs.org (manual) Distribution/Deploy OpenMRS 2.x/UAT02 - deploys to uat02.openmrs.org (manual) Distribution/Deploy OpenMRS 2.x/Demo - deploys to demo.openmrs.org (manual)

How exactly should it be? Do we need to clean up database of any of those servers? Which build deploys to qa01.openmrs.org? How to get https://help.openmrs.org/customer/portal/articles/1517999-using-test-servers updated?

Which servers do we need to create? Which ones can we delete?

(I need a step by step, otherwise I will end up doing the wrong thing)

In brief, this was my proposal:

  • devtestXX: dev playground, not CI
  • intXX: automatic deploys for automated testing*
  • qaXX: automatic deploys for manual testing*
  • uatXX: manual deploys for manual release testing*

*01 and 02 reserved for platform (core + REST) and distribution (refapp), respectively.

We can take these in bits; we’re actually closer than I thought.

The rest look okay (platform deploying automatically to int01.openmrs.org, UAT01/UAT02 manually deploying to corresponding servers, and demo manually deployed). I assume there’s a separate, automated process to reset the demo hourly.

When we get done shifting things around, there shouldn’t be any builds deploying to devtest servers.

My modification to the proposal was to specifically name the servers for platform and refapp as: int-platform int-refapp uat-platform uat-refapp

Distro builds should be deployed to int-refapp automatically, and to uat-refapp iff the UI tests pass. (Though I suppose it doesn’t truly need to be conditional.) CI already does this but we’d need to tweak the plans with renamed/switched servers.

Optionally the plans could be refactored, but the minimal amount of work really is just assigning the server names and making trivial mods to our build plans.

I guess I’m getting closer. That’s my current understanding:

So, when OpenMRS Core Master is triggered, it will:

  1. Deploy to int01/int-platform automatically.
  2. Run end-to-end tests on int01/int-platform
  3. If they pass, deploy automatically to qa01/qa-platform.
  4. Allow manual deployment to demo server and uat01/uat-platform

When Reference Application is triggered, it will:

  1. Deploy to int02/int-refapp automatically.
  2. Run end-to-end tests on int02/int-refapp
  3. If they pass, deploy automatically to qa02/qa-refapp
  4. Allow manual deployment to uat02/uat-refapp

Questions:

  1. Are we going as int01/int02/int0X or int-platform/int-refapp/int-extras-01? Both are good for me. I suppose we can start with the numbers and move it later.
  2. Where are the end-to-end tests of the platform?
  3. int02.openmrs.org and qa02.openmrs.org still don’t exist. Can any of the dev0[1-4] machines be destroyed/renamed?
  4. Why do we need both uat01/uat-platform and demo? Aren’t they the same thing? I’m not sure there’s a reset process on it. Maybe a cron process inside the machine, I don’t have easy access to it.
  5. If dev machines are not going to be deployed from CI, how developers are planning on using them? Do they need a key, or what?
  6. Which servers should Ebola be deployed to?

@cintiadr the Platform should not be quite as you describe it.

The platform should be driven by this github repo, and it should get its own plan(s) in the CI pipeline, similar to the Reference Application one. (Basically it’s openmrs-core + webservices.rest module. Eventually we may add a couple more modules.)

I can only answer a couple of your questions at the bottom:

  1. Where are the end-to-end tests of the platform?

We don’t have any. We should have some REST-based tests, I guess, since the platform eventually will not include a UI.

  1. Why do we need both uat01/uat-platform and demo? Aren’t they the same thing? I’m not sure there’s a reset process on it. Maybe a cron process inside the machine, I don’t have easy access to it.

demo should have the latest released version of the reference application (and its database should be regularly reset, but its code only updated on a release). uat would have the development version of code, and would be updated on every build.

  1. Which servers should Ebola be deployed to?

Ebola should just stay exactly as it is on devtest03.openmrs.org. No work is happening on that module at this point, so we don’t need to do any more deployments (and this should be disabled already) but we do want to keep that around for a while longer as a demonstration.

Hi there!

@cintiadr aologies, just woke up and saw this thread. So we’re currently using qa01.openmrs.org for hosting our FHIR work. We initially requested a QA server because the FHIR work was intended to be tested during a hackathon. Problem is, we configured apache to bypass security on this machine, and we need these configs to make the FHIR work run against the SMART Platform. Do you intend to destroy this setup? if so, i’d really love some grace period to copy over the apache configs so that I can replicate the current setup elsewhere :sunny: I’d hate to have to figure this out all over again…

Alright. So let me rewrite the platform

A plan for OpenMRS Platform needs to be created, and when it is triggered, it will:

  1. Deploy to int01/int-platform automatically.
  2. Run rest-based tests on int01/int-platform
  3. If they pass, deploy automatically to qa01/qa-platform.
  4. Allow manual deployment to demo server and uat01/uat-platform

When Reference Application is triggered, it will:

  1. Deploy to int02/int-refapp automatically.
  2. Run end-to-end tests on int02/int-refapp
  3. If they pass, deploy automatically to qa02/qa-refapp
  4. Allow manual deployment to uat02/uat-refapp

devtest03 show not be touched.

Questions pending:

  1. Are we going as int01/int02/int0X or int-platform/int-refapp/int-extras-01? Both are good for me. I suppose we can start with the numbers and move it later. @ryan has probably some opinion how hard it would be.

  2. int02.openmrs.org and qa02.openmrs.org still don’t exist. Can any of the dev01/dev02/dev04 machines be destroyed/renamed?

  3. If dev machines are not going to be deployed from CI, how developers are planning on using them? Do they need a key, or what?

  4. @darius If the platform won’t have a UI, would it make sense to deploy to int01/int-platform to run the REST-based test? Do we have those tests now?

  5. uat should be deployed manually or automatically? I believe @darius and @burke disagree here.

  6. Can I make deployment to demo automatic after a release of the platform? If the platform won’t have an UI, will it still be useful for a demo server? @ryan, do you know if the demo machine has a scheduled process for reset?

  7. To address @surangak comment - @ryan, should we rename the qa01 machine to qa03 instead? Or do you have a better idea?

  8. Also @ryan, do you think we should have the new machines as ubuntu 14? Or would it be too hard?

Yes this is right.

(Aside: I think it is a bit better if we always deploy to both int and qa, regardless of whether the int tests pass. But this is minor, and tangential to the main part of the work, so I’m fine leaving things as-is, i.e. “if they pass, …”)

  1. Are we going as int01/int02/int0X or int-platform/int-refapp/int-extras-01? Both are good for me. I suppose we can start with the numbers and move it later. @ryan has probably some opinion how hard it would be.

My personal vote is for int-platform/int-refapp. I don’t see why we would force people to learn “01” and “02” for fixed-purpose machines. But I’ll defer to @ryan’s opinion on this.

  1. If dev machines are not going to be deployed from CI, how developers are planning on using them? Do they need a key, or what?

I presume they will be given to devs on an as-requested basis, with some key-based process that exists, but is independent of the renaming we’re discussing here.

  1. @darius If the platform won’t have a UI, would it make sense to deploy to int01/int-platform to run the REST-based test? Do we have those tests now?

There is little real benefit to having all of int/qa/uat for the platform, but I think we should have a parallel structure for each distro anyway. We should write a couple of sentinel black-box tests of the REST functionality, but these tests don’t exist yet.

  1. uat should be deployed manually or automatically? I believe @darius and @burke disagree here.

UAT should be manual. I’m pretty sure that @burke and I agree on this.

  1. Can I make deployment to demo automatic after a release of the platform? If the platform won’t have an UI, will it still be useful for a demo server? @ryan, do you know if the demo machine has a scheduled process for reset?

Demo should have the reference application not the platform. Having a release of the refapp distro automatically deploy to the demo server is a nice-to-have, but since it only happens once every 6 months, I consider it a very low priority to automate.

This is correct. All non-production test machines are created upon request and assigned to the requester for the purpose stated in the request. They do not “exist” in our standard production server maintenance & change management processes. Details at: https://help.openmrs.org/customer/portal/articles/1517999-using-test-servers

The system at demo.openmrs.org is an exception because it is maintained as a public/advertising type of resource and as such it follows our standard change management processes, access control, monitoring, etc.

Any update on this? @cintiadr, are you actively working on it?

I agree that “int-platform/qa-platform/uat-platform” and “int-refapp/qa-refapp/uat-refapp” are preferable since it makes these longstanding resources more intuitive (easier to guess & remember).

Happy for infrastructure (@ryan) to paint this shed. :smile: … but, if we make these aliases now, then @cintiadr could script against them now and migrating the actual targets (e.g., moving the int-platform alias from devtest01 to int01) could happen independently.

If we can get a list of the aliases you want and their current name, we can set those up. For example: devtest01 -> int-refapp devtest02 -> qa-refapp

1 Like