The server we are currently using is largely being used by nexus (63.8(!) GB – Almost 64GB) and we do not have that much space on it. Builds will continue to fail. We are currently at 92.7% utilization of 117.99GB.
This is the cause of the recent Refapp builds failing. Nexus is the problem here right now.
We need to either:
a) move all the files to S3 and remove them from the server entirely – We could do this.
b) use bintray and discontinue nexus because this cannot continue. Logs aren’t taking up space, here, it’s Nexus.
Either way Nexus and all artifacts MUST be deleted from the server. This is not a negotiable here, what is negoitable is where they go before they’re deleted.
What is taking up most of the space within nexus? We (the OpenHMIS team) have been deploying all our released versions to the omrs nexus repo but have not thought about cleaning up old versions as we release new ones. Is there a way to trim older versions and free up some space?
I’d rather just nuke the whole thing to be honest. It’s actually a strain on our resources, so if you guys could use something else – that’d be great. If we migrate to bintray – we won’t have an issue…but right now it’s actually a strain.
Can we prune it? probably…but scripting this will be burdensome. This isn’t good at all. It’s time to retire nexus to be honest and move to Amazon S3 and/or bintray.
This is where we (the infra team) have to step in and not ask if you want it, but say this is how it has to be, here are your options. One application cannot hog over half the disk space like that. That’s just bad.
How would removing the nexus repository and moving to one of these other options affect module developers like us? We have actually already set up our own nexus repo but it mostly pulls from the omrs repo, with the exception of our SNAPSHOT builds.
I totally understand that you have to do what you have to do to keep the server going. I just want to make sure I understand the ramifications. Thanks for all your work to keep these things up and running!
If we are eating up all our disk space and we can’t get any more (though 120GB doesn’t seem like a particularly large amount, though I know Indiana U is donating it for free) then the current situation is certainly untenable. And if there are third-party solutions like Amazon S3 and bintray, it does seem silly to still be hosting our own. And one would hope that they’d make migration relatively straightforward.
That being said, handling the move incorrectly has the potentially to be (significantly) disruptive to the OpenMRS build pipeline, OpenMRS developers, and potentially other build pipelines as well like OpenHMIS, PIH and Bahmni. I don’t have free time this weekend to add my two cents on Amazon S3 or bintray, but will carve out some time to look into this in greater detail next week so hopefully I can weigh in more intelligently…
Unfortunately – builds are breaking as-is. The server is shared with Confluence(the wiki).
I want to do the move in a way that is not too disruptive, but it will not be pleasant either way sadly as things will break. I’d like to remove Nexus entirely and just use bintray or s3.
The less we have to maintain ourselves, the better.
I think we can address the issue in the following steps:
Disable proxying maven central. It should free up a lot of space on the server. I can do that, but need admin access to http://mavenrepo.openmrs.org/ It doesn’t require any changes from developers.
Start releasing to Bintray. It should be done during this week’s sprint.
Start deploying snapshots to Bintray (oss.jfrog.org). It shouldn’t be hard once 2. is done.
Migrate all openmrs released jars to bintray. It may require assistance from Bintray or we may need to write/find some script/tool to handle the migration ourselves.
The 4th step entails the most work. I would say that if the 1st step frees up enough space to satisfy us we can continue to use http://mavenrepo.openmrs.org as an archive for older versions and delay the 4th step.
I’m not sure if it entails downloading all artifacts… even still – one service cannot be hogging 30+GB of space. You’d have drop below a reasonable value…reasonable being ~10-15GB… 20 at most.
To step #3 that @raff mentions above, do we know if it takes anything to be one the “selected” open source projects (see first sentence in link below)?
@raff we would also have to update all the module poms to point to bintray instead of our maven repo, and then developers would have to updat as well, right?
Not ideal, but if #4 becomes too much work, is it possible to make a Maven Repo read-only and continue to host the older artifacts there? Or at least do that in the short-term while we figure out the best way to copy all the old artifacts over?
I’m not. I am however saying that a decision must be made within the next couple days. Figure out what we can afford and let’s make a decision. Nothing gets deleted without it being moved somewhere.
All decision and discussion will happen here, not on a call. Here.
Please, let’s use this thread to get back to basics.
Question #1: Do we actually need more space, or do we have things configured in a way that are incorrect? I see people on this thread saying that we don’t need the space, and I see people saying we need more.
Question #2: Is there an alternative approach to our Maven repo that we need to consider.
From my vantage point, this is not an issue that just cropped up magically. It’s been accumulating for some time obviously. If we need more resources, we can get them. This is not an issue of money as much as it is an issue of approach.
Let’s focus on an approach that requires the least amount of radical change and disruption to our community.
