Maintaining SNAPSHOT Maven dependencies

There is a topic fest this Thursday. Please bring ideas for dev forum topics there and we can get it on the schedule.

1 Like

In our opinion it is essential to be able to rebuild old versions of modules. LATEST is not good enough. I hope others will weigh in that direction too.

Maybe not essential, but I do agree with @mksd that being able to rebuild versions is important.

@cintiadr, thanks for digging further. If I recall correctly we started to import versions from the ref app pom in modules in order to be able to run module’s test suite against dependencies in the exactly same versions as declared in the latest build of the ref app. Let’s assume we have modules A, B and C depending on module D and we do a change in module D. We want to include that change in the Reference Application so we upgrade the distro to use module D in a SNAPSHOT version. Now when we build module A, B and C it is automatically updated to build and run tests against module D in the SNAPSHOT version.

At the same time we are aiming to run Reference Application builds against SNAPSHOT versions of modules and release modules with any changes before releasing the Reference Application.

Having that in mind, another solution would be to stop importing versions from the ref app in modules and instead configure CI to build modules with versions of dependent modules set to SNAPSHOT and also build the Reference Application against SNAPSHOT versions of modules. It feels simpler from the release perspective and achieves the same goal.

However, this approach will not work very well, if we actually include an older version of a module in the Reference Application. We will not have a way to test, if APIs are still compatible, but I feel it’s not that important. It may be caught by our ui tests or we can write proper API integration tests for such a case.

For what it’s worth, in our PIH distro, for PIH modules that aren’t part of the reference app as part of our build pipeline we use the versions plugin to make sure that we are always building against the latest version of a module. In bamboo, before the mvn clean install task we run “clean versions:update-properties scm:checkin -Dmessage='automatic update of mvn version dependencies' -U”.

We can configure the versions plugin to allow or not allow snapshot versions of module on a case-by-case basis. You can see a sample pom here:

Generally we’ve gotten to the point that we are depending on non-snapshot versions of most of the reference app modules, but some of the customization modules further down the dependency tree we leave in SNAPSHOT mode indefinitely. The two downsides to this is that:

  1. For modules for which we only depend on non-snapshot versions, we will only discover issues when a module is released (and therefore our build picks up the new version).
  2. The modules that we always use SNAPSHOT versions for never get official releases.

Perhaps if in the ref app we could use the versions approach but someone configure whether or not it allows snapshots on-demand… so by default it would allow snapshots, but then when we want to do a release switch to non-snapshot mode?

Wouldn’t it possible to just not remove the *-SNAPSHOT artifacts from the Maven repo then?

At least from an outsider (to the release process) point of view that would be better. Because for example if one depends on distro 2.3 and that somehow some dependencies on distro 2.3-SNAPSHOT are left down the line. Well, that is not the outsider’s business: if the Community “decides” that it is what distro 2.3 implies, then so be it. But since those older snapshots are removed from the Maven repo, it is sometimes just impossible to rebuild an older version of a project (even a fairly recent older version).

Thanks @mksd for bringing it back to this… I did indeed suggest we keep the snapshots of the reference application module around, at least until we come up with a better solution:

Any objections to setting this up? Any does anyone know how to configure the repo to set this up? :slight_smile:

Take care, Mark

2 Likes

I approve of making this change (just for the distro). I don’t know how to do it though.

-Darius (by phone)

Anyone know how to do this? @michael is this a help desk ticket?

I looked quickly and didn’t see an obvious way to do it–I may not have admin rights n the repo though.

Mark

Hi @bholagabbar @mogoodrich @cintiadr and everyone else involved in this thread. We would like to discuss this topic on the May 26th dev forum would you all be able to join the call?

1 Like

Since this is an engineering policy decision, I’d encourage us to keep the conversation here where everyone has the opportunity to participate and provide feedback, rather than doing it on a phone call with very limited participation.

Who is the person that has the final decision about how to proceed with version retention?

@michael, I don’t think there’s much further to discuss, as far as policy goes.

We have basically decided to try this quick fix. I will own this decision (as former RefApp lead, without any replacement at this point).

The question is, can we retain SNAPSHOT versions of org.openmrs.distro.referenceapplication in maven, without retaining snapshots of all the other artifacts? And who can configure this?

It looks like Neuxs currently has a configuration for a scheduled task of “Remove Snapshots from Repository” which runs hourly. Related, there is an “Empty Trash” job which runs daily.

Based on https://books.sonatype.com/nexus-book/reference/scheduled-tasks.html, the remove snapshot can only work on a repository level; in this case, our entire “Snapshots” repository at http://mavenrepo.openmrs.org/nexus/content/repositories/snapshots/. We have configured a minimum snapshot count of 1. I do not see any documentation for such a job that can exclude specific snapshots, without moving those specific files to a separate repository altogether.

However, there is also a “Remove Unused Snapshots From Repository” job available, that would restrict only to unused snapshots. I’m not sure how exactly this is measured, or if it would work for this scenario.

Scheduled tasks can be adjusted by any user with the “Nexus Administrator” role, which currently includes: Burke, Daniel, Darius, Fred Chen, Matt Blanchette, Mayank, Rowan Seymour, & Saptarshi.

Seems like it might make sense to:

  1. Make sure the “Remove Snapshots” task is not running on our “modules” repo

  2. Change the pom of the distro module to publish snapshots to the “modules” repo

I don’t know enough about Maven / Nexus to know if they’d be any issues with this.

This is the case … only runs against the “Snapshots” repo. Not sure if #2 is viable but hopefully someone can chime in.

I figured it might be as simple as in the pom changing:

        <snapshotRepository>
            <id>openmrs-repo-snapshots</id>
            <name>OpenMRS Snapshots</name>
            <url>http://mavenrepo.openmrs.org/nexus/content/repositories/snapshots</url>
        </snapshotRepository>

to:

<snapshotRepository>
        <id>openmrs-repo-modules</id>
        <name>Modules</name>
        <url>http://mavenrepo.openmrs.org/nexus/content/repositories/modules</url>
 </snapshotRepository>

But when I changed this and did a mvn clean deploy I got this message:

ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project referenceapplication: Failed to deploy artifacts: Could not transfer artifact org.openmrs.distro:referenceapplication:pom:2.4-20160503.182542-1 from/to openmrs-repo-modules (http://mavenrepo.openmrs.org/nexus/content/repositories/modules): Failed to transfer file: http://mavenrepo.openmrs.org/nexus/content/repositories/modules/org/openmrs/distro/referenceapplication/2.4-SNAPSHOT/referenceapplication-2.4-20160503.182542-1.pom. Return code is: 400, ReasonPhrase: Bad Request. -> [Help 1]

@darius is this still a topic you would like to discuss on a dev forum? If yes do we want to shoot for this month or next?

@jthomas, no this is not a dev-forum-worthy topic.

-Darius

1 Like

it came up during topic fest so thanks for the heads up. i will remove it from the list.

Does anyone have any further thoughts on #2 above? (See my last comment).

1 Like