MOD-52: Poll maven repository for new uploads

Ticket MOD-52 proposes adapting Modulus to poll maven for updates to modules. It sounds like, from a developer workflow, getting artifact deployed to Maven is the most convenient path. My concern is setting ourselves up for a model that doesn’t scale – i.e., creating a model that only works for people with write-access to the Maven repository and the only way for that to scale to the community is to make our Maven repository writable for anyone with an OpenMRS ID. Is that feasible?

Potential approaches:

  • Use Maven to upload modules & updates to modules to Modulus, allowing Modulus to take care of putting the artifact into the Maven repository.

  • Create a proxy that allows for “anonymous” additions to the Maven repository (basically, controlling where things can be written and maybe ensuring that you aren’t overwriting someone else’s work)

  • Configure the Maven repository to honor OpenMRS ID (i.e., Nexus LDAP configuration)

  • Make the space for module artifacts in the Maven repository world-writable (:shudder:)

  • Keep manually creating accounts in our Maven repository for anyone who wants to share a module (this doesn’t seem scalable)

Thoughts?

1 Like

I agree that requiring use of Maven or Nexus to publish a module in the OpenMRS Modules directory at modules.openmrs.org is a bad idea. IMHO, we want to make the process of advertas easy as possible, and I think we want modules.openmrs.org to represent as much of our ecosystem as possible. Every time we add “requirements” to module developers, we take a step in the opposite direction. (That’s not always bad, we just need to be conscious of doing so.)

Now, if Maven could somehow work with the same Modulus API that the web site uses to publish a module there, that’s certainly a parallel path that we could support.

On a related note, we previously tried to set up authentication to Nexus via Atlassian Crowd as a proxy to LDAP, which didn’t work well; and we found support for its Crowd connector to be horrible. We would need to make some architecture changes to our current LDAP stack (and in fact plan to do so) in order for Nexus to successfully authenticate with LDAP. This would happen in the next several months.

@darius, isn’t it sufficient that a developer invokes a “publish” command and the module ends up in the module repository & its artifact(s) in the maven repository? Why does it matter if your module’s POM has an OpenMRS-specific entry in it (it is an OpenMRS module, after all)?

Maybe the ultimate solution is none-of-the-above – i.e., if you want to share a module with the OpenMRS Community, you register your repository location (e.g., in a single SDK command) and both the Modulus and the Maven repository automagically get updated when you tag a new release. Is that magical thinking?

All I’m saying is that

  1. If I can keep doing “mvn release:prepare; mvn release:perform”, and this magically also gets the module repository updated with the newly-released version, this is literally a perfect solution for me. It does exactly what I want with no side effects or costs to me.
  2. This should not be too hard to implement. And there’s nothing wrong with allowing multiple solutions to the same problem.

A one-line SDK command is not something I’m personally interested in. I want to be able to use vanilla existing tools as much as possible.

The ultimate goal for me (though not for “random community modules”) is actually to have automation where JIRA or Bamboo can automatically do a release, and have it end up in both Maven and Modulus. But it’s important that the build happens once, and the exact same artifact ends up in both places.

1 Like

I don’t think we need one idea that solves it all. The solution I suggested is fairly straightforward to implement and can work as an option. You can either upload your module via Modulus UI or deploy it to the Maven repo and have Modulus pick it up or push it via REST or use SDK… whatever works for you.

1 Like

I agree. While I would prefer not to be limited by only having metadata that be pulled from a maven artifact, I agree with @darius that at this moment, the best thing to do is not to write a new maven plugin. In the interest of getting more developers interested in Modulus, I’d like it to affect their workflows as little as possible.

Right now, here’s the workflow I have in mind for a module developer that has Maven access.

  1. The initial upload of a Module (e.g. the first release) must be done manually on modules.openmrs.org.
  2. If the developer has maven access they can select to have Modulus watch for new releases on the Maven repo. (Essentially, just a checkbox in the upload form)
    • Maven access could be determined by membership in an LDAP group.
    • Modulus could show an example query so that the uploader knows it’s relating the correct maven artifacts with the module being uploaded.
  3. Modulus watches the recently deployed feed and matches artifact IDs from Maven with modules in its own database. When it finds new releases, it uploads them using metadata it can grab from config.xml and other sources. It notifies all maintainers of the completed upload via email.

If we can produce a good list of reasons why not to pull from our Maven repo, or what added flexibility our own maven plugin would provide, I’d be more interested in the alternative option.

The question I’m asking is:

We don’t have to have one way to do it, but we should at least have a way that works at scale.

1 Like

I’m not sure what is the status on this.

It doesn’t feel like there was a decision, but I was pretty much expecting a simple REST resource to upload, giving the artifact as a public URL.

So, I would be able to set up a deployment environment after a release:prepare release:perform (like this one), to upload a new version. I would pass the artifact in Nexus, but pretty much any developer around the globe could point to their own URL (their remote maven, apache service, whatever).

And for what it’s worth, as the release is a manual and conscious decision, it’s possible to retrieve the Bamboo user who kicked off the release build. Not sure how important is to update that information in modules itself, or if we should just link back to the original build which uploaded it.

About credentials, I would expect to have one user to deploy from ci.openmrs.org.

The good thing about this approach would be that modules.openmrs.org would be completely independent of any other service. Regardless of how people are releasing or deploying, as far they have a public URL for the omod, valid creds and any automation, they are free to use it.

1 Like

So…any updates here?

As I stated before, my stance is that Maven is the overwhelming consensus tool for Java dependency management, and we should go with this. Which means we should treat releasing a module to our maven repository as the “source of truth” about a module release.

But I’d accept any solution to this problem as long as we do something. Today my day was worse because I had to do steps beyond #1 here:

  1. automatically release appframework via bamboo (thanks to awesome work by @cintiadr)
  2. manually go to mavenrepo.openmrs.org and download the artifact I had released
  3. manually log into modules.openmrs.org and upload this artifact

If the module repo isn’t going to handle this automatically any time soon, then we will start scripting the CI builds to upload to the module repo… I assume there’s a REST resource for that.

It looks like Modulus’ REST api provides the methods to upload new releases for modules; however, it would probably take a little work to script it and work through the logistics (e.g., the script would need to supply module ID, OpenMRS ID of user posting module or a suitable proxy, etc.)

@darius, what would the integration with Maven look like? Would Modulus need to listen to a webhook or something similar?

If the implementation of autoreleasing from Maven seems straightforward, I’m not opposed to working on it.

If we go the route of scripting, I can help with that as well. The trickiest part is probably just getting the proper OAuth token to do the upload. The REST endpoints you’d need are all documented in Modulus’s API docs.

My first thought was:

  1. In the overall config for a module on modulus you can say something like: “[x] automatically get updates from maven”. (I don’t know if you’d need to explicitly give a URL like http://mavenrepo.openmrs.org/nexus/content/repositories/public/org/openmrs/module/appframework-omod or this should be purely by convention.)
  2. On a scheduled task the module repository looks at the URL + “maven-metadata.xml” and looks at /metadata/versioning/release in the XML. If this is a higher version than what’s currently the latest in modulus download it an add it to modulus as a new version.
  3. Do whatever dumb logic we have to do to change xyz-omod-version.jar to xyz-version.omod.

mavenrepo.openmrs.org is publicly-readable, so no credentials or permissions are needed for any of this.

I think this is a super handy thing for people using our Nexus installation.

That said, we should keep in mind (as I alluded to above) that we do not require OpenMRS module creators in the ecosystem to use our Nexus repository (nor do we require them to use our Modules directory site!) so this use case should never be on the default UX path but rather an opt-in feature. I think Darius mentions a good idea, to allow maintainers to specify a Nexus publishing path where Modulus could poll for updates. Of course, just because someone released something in Nexus, it doesn’t necessarily follow they want to advertise it yet on the Modules site. So that needs to be thought through a bit more.

On the other hand, Nexus has a plugin system for extending functionality and Modulus does not (yet?). Since we intended to create Modulus as a somewhat generic system with minimal dependencies, I feel more confident about leveraging Nexus plugins to push to Modulus, just as artifacts can be pushed to other Nexus repositories like Maven Central. It seems to follow how Nexus works a bit closer.

@darius, how many minutes do those extra steps require that made your day worse? While this work would save folks ___ time of work for each release, I personally I would give this a relatively low priority (probably after more UI improvements) , since releases are not a frequent activity and it’s not too arduous to upload a new release. But maybe I can be convinced otherwise if uploading a new release a real time sink.

The reference application contains 33 modules (if I counted right), and Mirebalais has a few more distinct ones. The specific steps don’t take much time, but it adds up to a lot of dumb tedious work when doing a release.

Cintia and I are actively working to better automate OpenMRS’s module release processes so that this is done by CI. I believe that having (an opt-in way) to make our module repo treat maven releases as the source of truth is both (1) easy, and (2) actually the right thing to do. And I’m happy to implement this or coordinate someone else to.

If you all don’t want to see this feature added to the module repo, then we’ll automate the release in some other way. I guess we’ll just write more lines of scripts to be run by ci.openmrs.org (making REST calls) when you release a module.

The alternative is not that we instead spend time working on the modulus UI, but rather that we do this in a way I think will take longer, and be a bit worse. (It’s fine, it will work that way too…)

So, I guess I’m asking: can we build the opt-in feature that I’m proposing in the very near future? Or is there opposition to this? If there is significant opposition, then solve our problem in another way and I won’t ask about this again.

The model I’m proposing is that you can opt into managing your module’s releases through maven. If you opt in to this (which all core-supported modules built by CI should do) then releasing to maven means you have released the module, period. If you don’t want “release to maven” => “release to module repo” then you shouldn’t opt into this feature.

I am annoyed by this. OpenMRS is a Java-based project, and it has put effort into building the OpenMRS Module Repository. We should be able to just use the overwhelming consensus solution for managing released artifacts in Java. We should not need to make modulus modular, and write a plugin to support Maven of all things. (wtf?)

I tried to ping @elliott by email 2 weeks ago, I’m not sure if he’s on leave. I wonder how hard it would be to get scripts to running rest calls against modules.openmrs.org. It already does everything via rest calls, the problem is I really don’t understand how to authenticate using a script.

Getting the possibility of doing things via rest from builds (or any other scripting automation) is a very powerful thing I’d like to experiment if possible. It would allow not only what we need right now (upload the modules), but several more advanced tasks we can do in the future (like verification of what was deployed, if everything is correct, update modules, check for updates inside a builds).

If I could authenticate, it would be a simple sed + curl to get the file deployed, and pretty much other implementers could do the same from their CI/CD tool, regardless of the building tool, the remote maven, or their CI tool. It would be clear from the build if the upload failed for any reason, and it keeps a nice UI saying when it was deployed successfully. It gives heaps of flexibility (also, you can choose when do deploy the module to modulus, after a green release build, or after a manual run, skipping milestones builds, so on).

Would that be possible?

Sorry for my absence @cintiadr, I’m a full-time student at this point and so I’m spending less time on OpenMRS-related things.

But sounds great. As for authentication, we currently have only implemented OAuth 2. Modulus doesn’t actually have any passwords in its DB, it instead uses OpenMRS ID as an authentication provider.

You could pretty easily script out the OAuth process via curl, with one exception: Modulus expects a redirect_uri that it can send the authorization token to, and from a script you don’t have a server running that can accept this token :slight_smile:

I’m playing with “out of band” redirect URIs as I’ve seen them in some other OAuth implementations, which would allow you to do the auth process without running a server. I’ll let you know what I figure out.

Created https://issues.openmrs.org/browse/MOD-84: “Out of band OAuth Requests”