# Addons and RefApp environments outage **URL:** \ **Category:** Infrastructure\ **Created:** [October 4, 2017, 8:34am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617 "2017-10-04T08:34:38Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [October 4, 2017, 8:34am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/1 "2017-10-04T08:34:38Z")
Hi everyone, This is becoming routine šŸ˜ƒ The following servers are unresponsive: - [addons.openmrs.org](http://addons.openmrs.org) - [addons-stg.openmrs.org](http://addons-stg.openmrs.org) - qa-refapp - demo - modules-refapp Iā€™ve contacted the provider. ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 14, 2018, 8:33am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/2 "2018-01-14T08:33:24Z")
This has happened again this weekend. I raised yet another ticket with the provider. ---
**Author:** ![jwnasambu](https://talk.openmrs.org/user_avatar/talk.openmrs.org/jwnasambu/32/13543_2.png) [@jwnasambu](https://talk.openmrs.org/u/jwnasambu)\ **Post date:** [January 14, 2018, 1:54pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/3 "2018-01-14T13:54:13Z")
@cintiadr any solution? ---
**Author:** ![reubenv](https://talk.openmrs.org/user_avatar/talk.openmrs.org/reubenv/32/3818_2.png) [@reubenv](https://talk.openmrs.org/u/reubenv)\ **Post date:** [January 14, 2018, 3:17pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/4 "2018-01-14T15:17:03Z")
A backup server with a different provider especially for light weight apps like add-ons could be one solution. Not sure if thatā€™s feasible ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 15, 2018, 2:39am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/5 "2018-01-15T02:39:29Z")
> [@jwnasambu](#): > > @cintiadr any solution? I thought it was clear by my message, I raised the ticket with the provider. Their whole datacenter is down, last update they sent was that they are having problems with backend storage. We are waiting for the provider to fix the problem on their end, thatā€™s why I raised the ticket with the provider. I cannot fix their datacenter for them. They are in the US, so Iā€™d expect them to be on the problem full hands when they wake up for their monday. > [@reubenv](#): > > A backup server with a different provider especially for light weight apps like add-ons could be one solution. Not sure if thatā€™s feasible. TACC appears to be slightly more unstable than the other datacenter, but so far Iā€™ve been giving some time for them to stabilize their datacenter, as Iā€™d rather not have _all_ our machines on the very same datacenter and have a single point of failure. Anyway, the datacenter is down, so I cannot even create a new VM. In all fairness, if I was to configure any sort of high availability, addons is pretty low on the list: > [@RFC: Production Tiers for our infrastructure](https://talk.openmrs.org/t/rfc-production-tiers-for-our-infrastructure/14832): > > Hi everyone, For those who arenā€™t aware, today we divide our infrastructure in testing, staging and production. Theory is production outages have more impact than testing outages, and hence we do deploy changes first to the lower environments. Also, outage response can be impacted by that, as testing machines are perceived as not being as important. The idea is that we want to group machine by their impact if thereā€™s a full or partial outage, allowing us to prioritise the work and calculate deā€¦ demo, modules-refapp and qa-refapp were considered more relevant. ---
**Author:** ![dkayiwa](https://talk.openmrs.org/user_avatar/talk.openmrs.org/dkayiwa/32/179_2.png) [@dkayiwa](https://talk.openmrs.org/u/dkayiwa)\ **Post date:** [January 15, 2018, 6:51pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/6 "2018-01-15T18:51:17Z")
I do not remember having seen these servers down for all this long. Are these the new server providers that we just moved to? Hopefully the provider is seriously looking into it! šŸ™‚ ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 15, 2018, 10:23pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/7 "2018-01-15T22:23:13Z")
> [@dkayiwa](#): > > I do not remember having seen these servers down for all this long. Are these the new server providers that we just moved to? Hopefully the provider is seriously looking into it! Neither do I. I donā€™t think it ever happened. I did expect more responsiveness from them today, what I assume itā€™s a regular working day. So, our new provider has two datacenters, Indiana and Texas. Idea would be to not put all our eggs in the same basket, so we could have half of the services on each (and an outage on the provider would take only half of our servers). Texas has been always a little bit more flaky than Indiana, so I kinda stopped putting new VMs there. Tonight I will recreate the refapp environments in Indiana, but I will have to trick terraform to ignore the existing VM šŸ˜• The biggest side effect is that if Indiana goes down, every single VM we have will go with it. In that case, always check [status.openmrs.org](http://status.openmrs.org), because thatā€™s not hosted by us. cc @burke and @paul ---
**Author:** ![mogoodrich](https://talk.openmrs.org/user_avatar/talk.openmrs.org/mogoodrich/32/3324_2.png) [@mogoodrich](https://talk.openmrs.org/u/mogoodrich)\ **Post date:** [January 16, 2018, 1:18am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/8 "2018-01-16T01:18:00Z")
> [@cintiadr](#): > > Neither do I. I donā€™t think it ever happened. I did expect more responsiveness from them today, what I assume itā€™s a regular working day. For what itā€™s worth, itā€™s a holiday here in the US today (Martin Luther King Day). Thanks for dealing with all this @cintiadr! ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 16, 2018, 8:34am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/9 "2018-01-16T08:34:38Z")
I recreated the refapp environments in Indiana. ---
**Author:** ![dkayiwa](https://talk.openmrs.org/user_avatar/talk.openmrs.org/dkayiwa/32/179_2.png) [@dkayiwa](https://talk.openmrs.org/u/dkayiwa)\ **Post date:** [January 16, 2018, 8:45pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/10 "2018-01-16T20:45:59Z")
Thank you so much @cintiadr šŸ™‚ Looks like the provider has not yet fixed this because addons is still down. ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 16, 2018, 9:30pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/11 "2018-01-16T21:30:56Z")
Yeah, the provider came back today, they are still working with the object store. I will recreate addons prod today. Maybe staging today, maybe tomorrow. ---
**Author:** ![jwnasambu](https://talk.openmrs.org/user_avatar/talk.openmrs.org/jwnasambu/32/13543_2.png) [@jwnasambu](https://talk.openmrs.org/u/jwnasambu)\ **Post date:** [January 16, 2018, 9:52pm UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/12 "2018-01-16T21:52:19Z")
@cintiadr thanks for your effort ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 17, 2018, 7:59am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/13 "2018-01-17T07:59:57Z")
I recreated addons (prd). I will do stg tomorrow. ---
**Author:** ![dkayiwa](https://talk.openmrs.org/user_avatar/talk.openmrs.org/dkayiwa/32/179_2.png) [@dkayiwa](https://talk.openmrs.org/u/dkayiwa)\ **Post date:** [January 17, 2018, 8:43am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/14 "2018-01-17T08:43:35Z")
That was fast!!! @cintiadr your awesomeness is more than am used to! šŸ’ ---
**Author:** ![cintiadr](https://talk.openmrs.org/user_avatar/talk.openmrs.org/cintiadr/32/8081_2.png) [@cintiadr](https://talk.openmrs.org/u/cintiadr)\ **Post date:** [January 18, 2018, 9:55am UTC](https://talk.openmrs.org/t/addons-and-refapp-environments-outage/13617/15 "2018-01-18T09:55:35Z")
I didnā€™t need to recreate lamu/addon-stg, they are back. Finally šŸ˜ƒ