Maintaining SNAPSHOT Maven dependencies

I ran into an issue with our dependency management within the ref app yesterday that unfortunately has no “easy” fix that is also “correct” that I know of. It’s not a blocker for our development right now and we have workarounds, but I wanted to share it just in case others run into the same issue:

We know we have a circular dependency issue within the reference app: in order to make it easier to control/update what versions of modules go into a particular refapp release, most of the modules in the refapp import the main reference app module pom and that’s where we define which versions of modules to use.

In order to make this work, released versions of modules are allowed to depend on snapshot versions of the reference app module. For instance, EMR-API 1.1 depends on Reference App 2.3-SNAPSHOT. However, since the 2.3 version of the reference application has now been released, 2.3-SNAPSHOT is no longer in the the OpenMRS maven repo. Therefore, if a new developer was to check out the 1.11 tag of EMR-API and try to build it, they would get an error because of being unable to locate Ref App-2.3-SNAPSHOT.

Furthermore, if the dev tried to build any module that depended on EMR-API 1.11, that build would also fail. I had this exact issue happen yesterday while trying to help someone get their development environment up and running again after a period of downtime. It was hard to track down where the bad dependency was coming from, and I was reluctant to dive in too deep because for all I knew solving this one issue could uncover another issue. My workaround was simply to zip up the refapp/2.3-SNAPSHOT directory in my .m2 directory and give it to them to install in their .m2 directory.

Long-term, we really need to rework our dependency management solution–but it is not an easy problem and might require a significant reworking. I know we’ve had many discussions about it in the past and no “magic bullet” has been uncovered.

We are not blocked on it now, but I propose that since we are releasing modules with snapshot dependencies on the referenceapp module, we at least start keeping all snapshot versions of the reference application module (but only that module) in the OpenMRS maven repo. This may prevent others from running into the same issue going forward.

Thoughts?

I think i have run into something similar, you’re right there is no clear solution but the work around is to clone the ref app distro project onto your machine, checkout the last revision before the version was changed, build the distro locally that way you get the 2.3-SNAPSHOT artifacts in your local maven repo.

The issue with that Wyclif is that the 2.3-SNAPSHOT itself depended on out-of-date snapshots (like platform 1.10.2-SNAPSHOT), so you’d have to check out and build a lot of modules to get what you need.

Again, not really a big deal for us, because we generally have the old refapp snapshot versions in our local repo or on our own bamboo server, but I could see it being an issue for others.

There has to be a revision in git where the distro project had no snapshot module dependencies because it is mandatory as part of the release process unless 2.3 was never released. Ideally the releasing manager must have had the distro at a revision where there were no module snapshots dependencies when releasing which i see here, in fact the alternaive could be to just checkout the RA 2.3 tag, change the version back to 2.3-SNAPSHOT and build it locally.

The pom file at the last revision where 2.3 was tagged can be seen here, it has no snapshots and its version is 2.3-SNAPSHOT, also in the 2.3 tag the pom file looks like this, again it has no snapshots but the version is set to 2.3 which you only need to change back to 2.3-SNAPSHOT and build the project to have the artifacts in your local maven repo, either option should be able to work.

Good point, Wyclif… you could just check out the stable 2.3 tag and then change it to 2.3-SNAPSHOT and rebuild. Won´t be obvious to a new dev, though…

Mark

As an aside, the last time I tried to use the distro, I came across a problem which I think I heard @darius mention in the youtube video discussing the bamboo scripts. In the uicommons module config.xml, for instance, ${uiframeworkVersion} is not substituted for a version number. So when you run the reference application, uicommons fails to see that the right version of uiframework has been loaded. (I only got as far as trying to load uicommons.) Unless somehow my setup is wrong, this is also a problem that also has to be addressed to enable the release of reference application 2.4 to be successful.

We should probably get this on a design forum at some point… just ran into this again today trying to get a dev environment set up for a new form developer… @darius @mseaton

And ran into this again today setting up en environment for a new developer… :slight_smile:

I say we bring this up on the dev forum? We can ask @jthomas if a slot is available. I too was setting up the reff app to work on an issue and ran into a similar problem.

A while ago @raff mentioned the same thing. Should we split the pom file on its own project/GAV? Should we have a pom with only released artefacts?

I’m not sure how to solve this, but using SNAPSHOT during releases looks now a pretty bad decision.

@cintiadr, I do think we should split the pom file on its own referenceapplication-versions project. The workflow for releasing Reference Application 2.4 would be as follows:

  1. Change all SNAPSHOT versions in referenceapplication-versions project to released or to-be released versions of artifacts
  2. Release referenceapplication-versions 2.4 project
  3. If any to-be released artifact depends on referenceapplication-versions 2.4-SNAPSHOT, change it to 2.4. Release any not yet released artifacts mentioned in referenceapplication-versions 2.4 project.
  4. Update referenceapplication-distro to use referenceapplication-versions 2.4 project.

I think it would eliminate issues we have. I might be missing something in the workflow so it would be good to verify it in practice before committing to that approach. What do you think?

I’m not sure, it still appears to be such a complicated workflow, not sure if there’s anything simpler.

Let me try to break down the problem.


Let me assume that reproducibility of older builds is less important than getting always the latest build working with latest code.

Modules should not rely on a parent pom 2.4-SNAPSHOT. We could either use LATEST or RELEASE. I Modules today are allowed to be released even if one of its dependencies is a SNAPSHOT. It was done this way to unlock different modules, so the modules could be released independently, not sure if it’s still applicable.


Here, let’s assume that you need older commits to keep old versions of artefacts. So then modules should use a released version when released (e.g. 2.4).


We can break reference-application and reference-application-versions, but what problem exactly are we trying to solve? Do we want to release modules tying them to specific released versions? Do we want to force dependent modules to be released before releasing this module? Or do we simply want everyone to be using the latest; regardless if it’s stable or not?

Do we want reference app to have a ‘stable’ branch? Is it too bad if we have a ‘released’ branch to be used as parent poms, and a snapshot for the integration tests?

I need more input. It has been a long time since we talked about why we needed this versions all together and discussed modules releases, I do believe quite a lot might have changed.

1 Like

Trying to think it again. We do have two problems.

One is the version of dependencies of modules. We want to be able to rebuild some old version of modules, with the correct version of its dependencies (right? Or we don’t care about older versions, and LATEST is good enough?). We want to compile and run the integration tests of the modules against the version which they will run in runtime (distro).

The second one is: we want to run selenium tests on every change.

Can we break these two (or three) problems? Somehow releasing modules more frequently? Or making them all part of the same repository, if they always change together. (you can merge repos without losing history)

There is a topic fest this Thursday. Please bring ideas for dev forum topics there and we can get it on the schedule.

1 Like

In our opinion it is essential to be able to rebuild old versions of modules. LATEST is not good enough. I hope others will weigh in that direction too.

Maybe not essential, but I do agree with @mksd that being able to rebuild versions is important.

@cintiadr, thanks for digging further. If I recall correctly we started to import versions from the ref app pom in modules in order to be able to run module’s test suite against dependencies in the exactly same versions as declared in the latest build of the ref app. Let’s assume we have modules A, B and C depending on module D and we do a change in module D. We want to include that change in the Reference Application so we upgrade the distro to use module D in a SNAPSHOT version. Now when we build module A, B and C it is automatically updated to build and run tests against module D in the SNAPSHOT version.

At the same time we are aiming to run Reference Application builds against SNAPSHOT versions of modules and release modules with any changes before releasing the Reference Application.

Having that in mind, another solution would be to stop importing versions from the ref app in modules and instead configure CI to build modules with versions of dependent modules set to SNAPSHOT and also build the Reference Application against SNAPSHOT versions of modules. It feels simpler from the release perspective and achieves the same goal.

However, this approach will not work very well, if we actually include an older version of a module in the Reference Application. We will not have a way to test, if APIs are still compatible, but I feel it’s not that important. It may be caught by our ui tests or we can write proper API integration tests for such a case.

For what it’s worth, in our PIH distro, for PIH modules that aren’t part of the reference app as part of our build pipeline we use the versions plugin to make sure that we are always building against the latest version of a module. In bamboo, before the mvn clean install task we run “clean versions:update-properties scm:checkin -Dmessage='automatic update of mvn version dependencies' -U”.

We can configure the versions plugin to allow or not allow snapshot versions of module on a case-by-case basis. You can see a sample pom here:

Generally we’ve gotten to the point that we are depending on non-snapshot versions of most of the reference app modules, but some of the customization modules further down the dependency tree we leave in SNAPSHOT mode indefinitely. The two downsides to this is that:

  1. For modules for which we only depend on non-snapshot versions, we will only discover issues when a module is released (and therefore our build picks up the new version).
  2. The modules that we always use SNAPSHOT versions for never get official releases.

Perhaps if in the ref app we could use the versions approach but someone configure whether or not it allows snapshots on-demand… so by default it would allow snapshots, but then when we want to do a release switch to non-snapshot mode?

Wouldn’t it possible to just not remove the *-SNAPSHOT artifacts from the Maven repo then?

At least from an outsider (to the release process) point of view that would be better. Because for example if one depends on distro 2.3 and that somehow some dependencies on distro 2.3-SNAPSHOT are left down the line. Well, that is not the outsider’s business: if the Community “decides” that it is what distro 2.3 implies, then so be it. But since those older snapshots are removed from the Maven repo, it is sometimes just impossible to rebuild an older version of a project (even a fairly recent older version).