We’re getting closer to a better O3 release process with huge thanks to folks like @ibacher and @raff, and the practical QA findings/tech implications uncovered by folks like @jnsereko. Thank you!!
Here’s a first draft diagram of our O3 Release Pipeline Process 2022 - @ibacher@raff@jnsereko@dkayiwa@dkigen@jayasanka & others: could you have a look at this and see if I’m on the right track?
The end result of this release process will hopefully be really helpful to groups like OHRI & Ozone, who are directly trying to leverage O3 Reference apps, in whole or in part.
Thanks SO much for working through this. This a terrific start to a difficult problem!
If we are using @latest for auto-building on dev3, do we need to trigger a build on every commit to an esm or only when there is a change to @latest?
How does a developer verify things are going to work before releasing a new (latest) version of an esm? I’m assuming a developer would typically use the latest stable build (the dev3 image) with their import map pointing to the bleeding edge or upcoming release (e.g., @next) version of any esm(s) they are actively working on. If we’re testing commits to the esm during development (between new releases of the esm) using Bamboo, we would need to set up a similar environment (dev image using @latest with target esm using @next or simply HEAD of the main branch). Otherwise, developers would have to release a new version before it could be tested in CI.
In general, what I’ve read about continuous delivery best practices is that we want to lean on automation as hard as we can for setting up a development environment and testing. Paradoxically, the more (manual) barriers/bottlenecks we put into the release process, the less stable things could become, since it makes releasing cost more and, in turn, more changes are happening between releases and the longer it takes to get fixes into a release. If we can achieve a process where releases could happen multiple times per day, then problems can be fixed much more quickly.
Feature toggles could be used for changes in application behavior (e.g., instead of replacing old behavior with new behavior, the new behavior is added alongside existing behavior and substituted only when toggled on via a parameter or frontend config setting). If we haven’t already, it might be worth experimenting with this approach and introducing a best practice for how & where to introduce toggle setting in a way the end user could manipulate them.
We’re actually using next, so every commit to an ESM repo is effectively an update. Basically the rule is that every merged commit to an ESM should show up on dev3 as soon as possible. In other words, dev3 is a real continuous deployment environment for frontend changes.
In terms of the latest release, I think there’s a lot to be desired. Currently, every monorepo is versioned as a single entity. For some cases (e.g., esm-framework contains a number of components, but is generally one library from the perspective of a frontend), this makes sense. For most components, it doesn’t. But it leaves us with the only sensible release policy being versioning everything at the point we’re happy to push it to the demo environment.
Is @next the same thing as HEAD or SNAPSHOT (i.e., latest commit)?
From what I’ve heard, it sounds like the combination of the bleeding edge (latest commit) of every repository is what makes dev3 unstable. If every commit shows up on dev3, it could raise the barrier for commits (encouraging devs to hold back commits), make it harder for devs to share development on work in progress prior to it being ready for wider exposure (e.g., dev3), and force devs to test their work in progress against everyone else’s – possibly unstable – work in progress.
Does every commit get published to npm? And is it possible to develop against an ESM without publishing it to npm? It seems like we could benefit from distinguishing between the latest commit and what goes to the dev3 environment. Could this be as simple as avoiding pushing every commit to npm and having a distinct “publish to npm” step? If a handful of developers working on an ESM can share their work in progress by building directly from the ESM repo (without publishing to npm), then they (and even GitHub actions ± CI) could test their work in progress before it ends up on dev3 and they could promote their work to dev3 simply by publishing changes to npm. This would allow dev3 to serve up the latest published development ESM without having to make everyone work against everyone else’s raw work in progress.
In a way, but the instability is sort of essential to the whole CI / CD process. That is, if we don’t have an environment that has all the changes as they are made, we lose the advantages of CI and only catch cross-cutting bugs when we decide to integrate. It’s true that this can (and arguably should) discourage commits to the main branch until the code is stable, but that’s why Git’s cheap branching and PRs are a kind of solution here, that is the work can be shared in-progress and even used by multiple devs, but won’t be committed until the feature itself is relatively stable. Put more simply, we’re not committing work-in-progress to main but to feature branches.
For community repos, yes, but as a pre-release.
Yes, although it’s not possible to use the tooling to include a package in a distribution without publishing it to a NPM repository (just like it’s difficult to include a module in a distribution without it being published to a Maven repo).
@burke i strongly feel that we still need a server that has the latest commit of the repositories. How shall we get closer to continuous deployment when we even do not have such an environment. This will help us get towards the culture of having each commit being a candidate for releasing. I think the hesitation towards this is caused by the lack of tests which fail when a commit breaks something.
One can use test3 if they are not interested in the latest commit.
It is easier to fix bugs which manifest on commit rather than later.
I completely agree. Dev3 for unstable, new, cutting edge stuff; Test3 for stuff that devs have declared to be stable and ready for release (we’re not there yet in terms of our pipeline set up, but I’d like to be).
They would use dev3 to do this. Sorry I got it wrong in my diagram - I’ve updated it to clarify that the current build process uses @next for the build, not @latest.
Where do we go from here? Idea: Test3 = @latest?
@ibacher what if the build setup for Test3 pulled in @latest? Meaning we would say to devs, “when your feature is ready and you’ve tested it in dev3 (integration), bump it up on npm to @latest!” Is that possible for a contributing dev to do? Or maybe they just notify one of the Global Product Support Team members to do an npm release for that esm to @latest?
Then, we would basically be using npm’s @next/@latest as a blunt proxy for feature flags.
@dkayiwa it makes my heart so happy to see you mention feature flags/toggles. This was completely mission critical to my SaaS experience where we did full production releases 1-2x/week, every week. Are we currently using toggles anywhere?
However, what worries me about them in the OpenMRS case, is that they do take oversight and maintenance - we would have to keep track of them, remember to adjust each one, and regularly review them for left-behind-stragglers that need clean up. As the O3 committer community grows I’m not sure we’d be able to manage that granular complexity easily. Which is why @next vs @latest sounds so attractive to me.
But maybe I’m missing something. Curious to hear what you think @dkayiwa & @ibacher & @dkigen!
We reviewed this on the squad call today. 2 key takeaways:
@next makes sense as the esm-version-source for dev3, but @latest makes most sense for o3, not test3, because marking an esm with @latest implies to the whole world on npm that the app is stable / ready for the real world. And this is not necessarily true for all stuff in test3.
I’d like devs to indicate “I have tested X feature in dev3 and it works as expected” somehow - maybe through a label of some kind? Then test3 could pull in those specific updates; things that are basically halfway between @next and @latest.
Any ideas on what that “dev3-approved” label could be, technically?
As we (ICRC) oftenly use the @new version we could help on this point ( help on Cypress part, playwright…). How can we start on this point ?
FYI, we are using RobotFramework ( Selenium inside) to test our distro: we start a temporary openMRS instance, wait for it to be ready then launch tests ( for our test, we create a dedicated user, a patient, start a visit,…). But not sure it could help in this case
So nothing is supposed to be tested directly on dev3, but on the same images we’re publishing as dev3 / test3. The diagram is still somewhat aspirational, but so far focused on gates 1 (run e2e tests on PR) and 2 (run full e2e suite after merge). Gate 3 is more focused on releasing to test and some work needs to be done to enable that stream to be separated from the dev stream. The current 3.x tests we have live in this repo, however, as part of the work for gate 1, we’re aiming at decomposing tests along similar lines that the 3.x repos are broken down so that tests for the patient chart live alongside the patient chart, etc.
with huge thanks to folks like @ibacher and @raff, and the practical QA findings/tech implications uncovered by folks like @jnsereko. Thank you!!