Addons and RefApp environments outage

Hi everyone,

This is becoming routine :smiley:

The following servers are unresponsive:

I’ve contacted the provider.

1 Like

This has happened again this weekend.

I raised yet another ticket with the provider.

@cintiadr any solution?

A backup server with a different provider especially for light weight apps like add-ons could be one solution. Not sure if that’s feasible

1 Like

I thought it was clear by my message, I raised the ticket with the provider. Their whole datacenter is down, last update they sent was that they are having problems with backend storage.

We are waiting for the provider to fix the problem on their end, that’s why I raised the ticket with the provider. I cannot fix their datacenter for them. They are in the US, so I’d expect them to be on the problem full hands when they wake up for their monday.

TACC appears to be slightly more unstable than the other datacenter, but so far I’ve been giving some time for them to stabilize their datacenter, as I’d rather not have all our machines on the very same datacenter and have a single point of failure. Anyway, the datacenter is down, so I cannot even create a new VM. In all fairness, if I was to configure any sort of high availability, addons is pretty low on the list:

demo, modules-refapp and qa-refapp were considered more relevant.


I do not remember having seen these servers down for all this long. Are these the new server providers that we just moved to? Hopefully the provider is seriously looking into it! :slight_smile:

Neither do I. I don’t think it ever happened. I did expect more responsiveness from them today, what I assume it’s a regular working day.

So, our new provider has two datacenters, Indiana and Texas. Idea would be to not put all our eggs in the same basket, so we could have half of the services on each (and an outage on the provider would take only half of our servers).

Texas has been always a little bit more flaky than Indiana, so I kinda stopped putting new VMs there.

Tonight I will recreate the refapp environments in Indiana, but I will have to trick terraform to ignore the existing VM :confused:

The biggest side effect is that if Indiana goes down, every single VM we have will go with it. In that case, always check, because that’s not hosted by us.

cc @burke and @paul

For what it’s worth, it’s a holiday here in the US today (Martin Luther King Day).

Thanks for dealing with all this @cintiadr!

I recreated the refapp environments in Indiana.

Thank you so much @cintiadr :slight_smile:

Looks like the provider has not yet fixed this because addons is still down.

Yeah, the provider came back today, they are still working with the object store.

I will recreate addons prod today. Maybe staging today, maybe tomorrow.

@cintiadr thanks for your effort

I recreated addons (prd).

I will do stg tomorrow.

1 Like

That was fast!!!

@cintiadr your awesomeness is more than am used to! :bouquet:

1 Like

I didn’t need to recreate lamu/addon-stg, they are back.

Finally :smiley: