Addons and RefApp environments outage

[cintiadr (Cintia Del Rio)]

Hi everyone,

This is becoming routine

The following servers are unresponsive:

• addons.openmrs.org
• addons-stg.openmrs.org
• qa-refapp
• demo
• modules-refapp

I’ve contacted the provider.

[cintiadr (Cintia Del Rio)]

This has happened again this weekend.

I raised yet another ticket with the provider.

[jwnasambu (Juliet Wamalwa)]

@cintiadr any solution?

[reubenv (Reuben Varghese)]

A backup server with a different provider especially for light weight apps like add-ons could be one solution. Not sure if that’s feasible

[cintiadr (Cintia Del Rio)]

@jwnasambu

@cintiadr any solution?

I thought it was clear by my message, I raised the ticket with the provider. Their whole datacenter is down, last update they sent was that they are having problems with backend storage.

We are waiting for the provider to fix the problem on their end, that’s why I raised the ticket with the provider. I cannot fix their datacenter for them. They are in the US, so I’d expect them to be on the problem full hands when they wake up for their monday.

@reubenv

A backup server with a different provider especially for light weight apps like add-ons could be one solution. Not sure if that’s feasible.

TACC appears to be slightly more unstable than the other datacenter, but so far I’ve been giving some time for them to stabilize their datacenter, as I’d rather not have all our machines on the very same datacenter and have a single point of failure. Anyway, the datacenter is down, so I cannot even create a new VM. In all fairness, if I was to configure any sort of high availability, addons is pretty low on the list:

RFC: Production Tiers for our infrastructure

Hi everyone, For those who aren’t aware, today we divide our infrastructure in testing, staging and production. Theory is production outages have more impact than testing outages, and hence we do deploy changes first to the lower environments. Also, outage response can be impacted by that, as testing machines are perceived as not being as important. The idea is that we want to group machine by their impact if there’s a full or partial outage, allowing us to prioritise the work and calculate de…

demo, modules-refapp and qa-refapp were considered more relevant.

[dkayiwa (Daniel Kayiwa)]

I do not remember having seen these servers down for all this long. Are these the new server providers that we just moved to? Hopefully the provider is seriously looking into it!

[cintiadr (Cintia Del Rio)]

@dkayiwa

I do not remember having seen these servers down for all this long. Are these the new server providers that we just moved to? Hopefully the provider is seriously looking into it!

Neither do I. I don’t think it ever happened. I did expect more responsiveness from them today, what I assume it’s a regular working day.

So, our new provider has two datacenters, Indiana and Texas. Idea would be to not put all our eggs in the same basket, so we could have half of the services on each (and an outage on the provider would take only half of our servers).

Texas has been always a little bit more flaky than Indiana, so I kinda stopped putting new VMs there.

Tonight I will recreate the refapp environments in Indiana, but I will have to trick terraform to ignore the existing VM

The biggest side effect is that if Indiana goes down, every single VM we have will go with it. In that case, always check status.openmrs.org, because that’s not hosted by us.

cc @burke and @paul

[mogoodrich (Mark Goodrich)]

@cintiadr

Neither do I. I don’t think it ever happened. I did expect more responsiveness from them today, what I assume it’s a regular working day.

For what it’s worth, it’s a holiday here in the US today (Martin Luther King Day).

Thanks for dealing with all this @cintiadr!

[cintiadr (Cintia Del Rio)]

I recreated the refapp environments in Indiana.

[dkayiwa (Daniel Kayiwa)]

Thank you so much @cintiadr

Looks like the provider has not yet fixed this because addons is still down.

[cintiadr (Cintia Del Rio)]

Yeah, the provider came back today, they are still working with the object store.

I will recreate addons prod today. Maybe staging today, maybe tomorrow.

[jwnasambu (Juliet Wamalwa)]

@cintiadr thanks for your effort

[cintiadr (Cintia Del Rio)]

I recreated addons (prd).

I will do stg tomorrow.

[dkayiwa (Daniel Kayiwa)]

That was fast!!!

@cintiadr your awesomeness is more than am used to!

[cintiadr (Cintia Del Rio)]

I didn’t need to recreate lamu/addon-stg, they are back.

Finally