JIRA license error!

No, we use pingdom.com which sends alerts to both helpdesk and pagerduty.

And this tool can restart JIRA automatically? Maybe it thinks JIRA is down but itā€™s just slow.

1 Like

No ā€“ all this tool does is literally visit the site and click around and if it canā€™t complete the tasks we defined, then it alerts us via pagerduty and also sends a message to helpdesk.

I do have a JuiceSSH snippet to restart JIRA on my phone and a similar one for ID dashboard (both of which wind up being quicker than me manually doing it by hand).

Maybe the time-out to complete the task is short and thinks JIRA is down when it isnā€™t ?

No, itā€™s down when we get the alert.

Next time you receive the alert please check if JIRA is really down, and the logs to discover the root reason (out of memory or whatever) otherwise itā€™ll keep happening every now and then.

My theory is that itā€™s not the heap that gets exhausted but Permgen memory, possibly because too many plugins (some of them throw exceptions because they are not compatible with our JIRA version, all is in the logs).

1 Like

Yeah, I havenā€™t been on call. Often times it goes down when Iā€™m not near my laptop, so restarting it is the easiest thing to do. If Iā€™m asleep, Iā€™m not gonna get out of bed to check the logs.

Interestingly, Confluence has been running 24 days, 12 hours, 27 minutes, 17 seconds

But maybe because it has LOTS of memory > JIRA (2x permgen, 4x heap)

1 Like

Should create a small elasticsearch in digital ocean and extract those logs for doing some analytics.

@chagara, I actually think we should spin up an instance of Logstash that we can use to aggregate all our logs for analysis.

What do you guys think?

1 Like

1 Like

Good idea. Depending how much logs it is you can do everything in one VM. Im currently working on this for my cloudera cluster. Let me know and i can help in any way.

IMO with the System Tools already available in JIRA could be enough. Thereā€™s a log analysis that already provides links to KB articles.

The idea is to make it easier to inspect the logs of all our applications. It might also be safer to give users a Logstash account than to give them SSH access to our boxes.

awesome :slight_smile: whatever works. I was just suggesting that I guess for those that do not have SA access to Jira and look at the errors without having to dig into the server or the app. Use Kibana or something similar to view them.

But unless Atlassian has some Kibana/LogStash artifacts that can analyze the logs and extract meaningful info from them, it must be done from scratch and I donā€™t think itā€™s easy at all (e.g. determine that a class cast exception happens because the GIT plugin is outdated, or a bug that is solved in version x.y.z).

Since Atlassian products are based on java then the logs can be extracted but, I do agree the configuration takes a little bit to get it to work but, it can be done.

https://www.elastic.co/guide/en/logstash/current/input-plugins.html https://www.elastic.co/guide/en/logstash/current/output-plugins.html

It seems like two different things are useful:

  1. Actually look through Atlassianā€™s logs and their recommendations, and take some actions

  2. Set up ElasticSearch+Logstash+Kibana for logs across much more of our infrastructure.

For #2, one blocker is that ā€œOpenMRSā€ canā€™t spin up a server for this. (Though, any individual interested person is welcome to contribute a $5-10/month digitalocean server for this!)

Can we seriously not afford $5 more a month @darius? We already have servers running on DigitalOcean.

I have no insight into the budget. I just know that every time I have asked for a server in the last couple years the answer has been that we donā€™t have any to spare.

And Iā€™m currently hosting the scrum logging bot on digital ocean myself, since asking to put it on OpenMRS hosting didnā€™t go anywhere.

(Itā€™s actually a totally legitimate option for someone to set this up in their own digital ocean instance, if they are in a position to donate $60/year of server hosting to OpenMRS.)

-Darius (by phone)