A CI build that has frozen

I manually did run this build yesterday. But 23hours later, it is still queued. Does any one know what is going on? @cintiadr, @r0bby, @raff and others? https://ci.openmrs.org/browse/TRUNK-LATEST-260

Surely it’s related to this: Bamboo agent upgraded, watch out for unexpected behaviours - #6 by cintiadr

I just changed this build’s configuration to use openjdk7, and it is now running. (Hope it passes!)

@cintiadr is there a global way in bamboo that we can remove no-longer-supported options from whatever power this dropdown:

…and be notified of any existing plans that can no longer run?

1 Like

PS- the build did run, but it failed because of the same error that was there in prior builds (something about a timezone). I’m not taking ownership of this failing build.

Thanks @darius for triggering the build. I had switched it back to JDK 7. Can you help me switch again like you had done to openjdk-7?

@dkayiwa are you saying that we can’t build 1.9.x on openjdk (even though we’re building 1.10.x on openjdk)?

My strong preference would be to make sure that our code can run on openjdk.

If that’s not possible for some reason, you should talk to the infra team about what java versions are available on CI.

Don’t tag me :slight_smile: I’m only around to ensure that the **** doesn’t fly and ID dashboard is (somewhat) stable.

A few things:

  • If your job is queued for a long time, look at the queue https://ci.openmrs.org/currentActivity.action . If the requirements don’t match any agent, it will show up there.
  • There’s alerting when that happens, but we only see it when we login to helpdesk :slight_smile: I usually action on the alert and fix the requirements of the build.
  • JDK 7 and openJDK 7 are the same, we don’t have oracle JDK installed. How do you know? :slight_smile: Click on any agent in cog-> Agents, and see the capabilities: https://ci.openmrs.org/admin/agent/viewAgent.action?agentId=50495494 You’ll see the full path of the JDK, which happens to be the same
  • I will kindly ask to never rely on the update version (Use always JDK 1.7, instead of JDK 1.7.0_111. Unfortunately that capability is automatically created by Bamboo, but as soon as we update machines that JDK won’t be available anymore…
  • The root cause of the problem was that this build was relying on an specific update, and unfortunately the agents with that JDK update were offline.

In order to remove something from that combo, it’s usually a matter of deleting capabilities from the agents. Most of them I can control via puppet, except for some JDK ones (like the jdk update one) - and I do that from the Bamboo UI. To make matters worse, the JDK update capability is not deleted when we upgrade the JDK.

If you click on the capability in agents’ Bamboo page (e.g. https://ci.openmrs.org/admin/agent/viewAgent.action?agentId=50495494), you can see which agents have it, and which jobs rely on it (e.g. https://ci.openmrs.org/admin/agent/viewCapabilityKey.action?capabilityKey=system.jdk.JDK+1.8) . It doesn’t show any jobs using it as a variable, though.

Other good tool is https://ci.openmrs.org/admin/agent/viewAgentPlanMatrixWizard.action. All our agents are the same, so all your jobs should be able to run in all agents.

I will review the JDK updates we have available now and clean older updates capabilities today (as in, I’m going to confirm the JDK exact versions installed on agents, and delete using bamboo UI any update which is not installed anymore).

Thank you so much @cintiadr for the fully detailed explanation! :bouquet: