Creating a build pipeline for OpenMRS 3.0

Yeah, it’s totally safe, as long as it isn’t pointing to the repo rubygems-proxy.torquebox.org (we use that for SASS in some of the UI modules; see here).

@burke any updates? Are we any closer to having two environments, dev vs production/demo, for 3.0?

@mksrom is the spa module supposed to be included in pom.xml? When I try to bring it up, it looks like resources are expected to be served from /openmrs/spa/ and I assume that path is designed to be handled by the spa module, right?

The MFs are served via the Nginx server. not the spa module.

The issue you encounter has been reported to me already but only appear for some and not others, maybe OS dependent. which OS do you run?

Immediate workaround:

Could you replace:

with:

npx openmrs build --spa-path /ui --api-url /openmrs --target $1

and rebuild again?


Longer explanation:

My idea was to keep the packaging of the application agnostic of the way it’s deployed.

However, the build of the MF project needs to be informed of some deployment-specific variables. And those vars are:

  • apiUrl
  • spaPath

Such variables must be passed to the npx openmrs build command, meaning that a package of the distro would become deployment specific!!

To get around this, I am setting the value to be environment variables instead:

build.sh

And substitute the vars upon running the project:

docker-compose.yml

Problem is that in some environments, --spa-path \${SPA_PATH} substitutes the variable with it’s current value (which is null on the developer machine) and therefore ends up being the default /openmrs/spa.

1 Like

Thanks @mksrom. I’m running on macOS Big Sur. Hardcoding the values worked. So did this:

npx openmrs build --spa-path \${SPA_PATH:-/ui} --api-url \${API_URL:-/openmrs} --target $1

What do you think about using default values? :slight_smile:

Are there requirements for the host (maven version, npm/npx version)? We’ll want to make sure our CI environment meets those requirements.

Not really.

I made sure that the most recent Maven works (by using HTTPS repos).

As for npm, on the machine I’ve tried it on, npm -v was 6.14.4

And any recent Docker and Docker Compose version will do.


My initial goal of this Distro Ref App 3.x project was to suggest a standard packaging for OpenMRS 3, not so much how to run such package. So the Docker project attached is really optional and there to show an example of how we can consume the packaging of OpenMRS.

With that in mind, I want the output ZIP file of the distro to be deployment independent. So that anyone can choose to deploy it the way he/her wants. Maybe with containers, maybe bare servers or anything.

Those 2 vars (apiUrl and spaPath), they make the package deployment dependent! :angry: :smiley:

So that’s the reason why we can’t hard code them, nor use their default value, as we don’t know what their value should be until we know how the app will be deployed.

So basically, that’s the job of the deployment project to provide those values.

In our case, the Docker project provides them in the docker-compose.yml

Does that make sense?

@mksrom my docker container has the correct environment values:

$ docker-compose exec frontend bash -c 'env | grep -iE "SPA|API"'
API_URL=/openmrs
SPA_PATH=/ui

I think the problem is package/scripts/build.sh is being executed as part of the mvn clean package (i.e., executed during the build within the host environment) before docker is started, so setting these environment variables in the docker-compose doesn’t help the npx command that was run before docker-compose is invoked.

So, either these variables need to also be provided during the build step or we need to perform the build within the docker-compose stack. This worked for me:

SPA_PATH=/ui API_URL=/openmrs mvn clean package
cd run/docker/
docker-compose up -d --build

Well, OpenMRS 3.0 doesn’t yet successfully get past the login, but at least the frontend appears to be getting loaded now. :slight_smile:

Looking through the docker-compose.yml, I don’t think we’ll need the proxy service, since that is already provided by the OpenMRS infrastructure. But we will still need to move the contents of distro/ and properties/ folders into a docker image.

Ideally they are supposed to be kept as environment variables during the build, so they can be substituted only at run time.

But anyway, we’ve made some refactoring on the project and it doesn’t look like you’re using the latest changes there, as we are now using the OpenMRS SDK to package the distro.

In this new version, the apiUrl is hardcoded at build time though:

Which again, is incorrect as we do not know those values for certain when packaging the app. Only know them upon running it.

In the case of deploying from CI to our infrastructure, the build environment and run time environments are different (build on CI, run via docker-compose on infrastructure host machines).

In the case of infrastructure deployment, we should not alter the host environment; rather, environment setting should be pulled from a configuration file (e.g., .env) just as Docker does.

And, from what I see for 3.0, these variables are being used both at build time (running npx as part of the maven packaging process) and for run time.

Ah. So, you’re saying that – unlike the current state of the 3.x branch of refapp distro – the npx command in the build.sh script that runs during mvn clean package should not need to supply these settings at all, since they should be determined within the app from environment variables at run time, correct?

Exactly that. That’s why initially I wanted them to just be set to the strings "$API_URL" and "$SPA_PATH". And provide their actual values only when running the Docker Compose project. See

But that’s not the case anymore. Right now, they are just hardcoded at build time. And that’s not correct

So, it looks like we only need to get the contents of the distro/ folder into container(s) so a docker-compose.yml can download everything needed from Docker Hub (including these resources) and provide the resources to the openmrs and frontend services.

@ibacher did @mksrom say that you had already done something along this line? I don’t know if creating a docker image from scratch to simply copy the distro files and provide them for the other containers would be considered a best practice in Docker-land… or if we just want to copy these files into the openmrs & frontend Docker containers (which would make those containers less re-usable).

So what I did is a bit messy, but hopefully manageable. Basically if you look here, there are two Docker images, one of which is expected to use volumes and one (the Dockerfile-embedded one) that just embeds the relevant files in the same place. (There’s a similar structure for the ui image that serves the frontend).

I think it looks like we just need to add a docker-compose-embedded.yml file and we could have something that can be deployed to the OMRS infrastructure.

Adding one more consideration: right now, openmrs-spa.org gets updated more or less “live” by updating an importmap located in DigitalOcean’s CDN. If we convert to the Docker images, the importmap is only setup to serve particular versions of the frontends, since the build generates the import map.

@florianrappl Is there any easy way we can use the standard build / assemble commands but continue leveraging the existing importmap?

Could we just change the distro.properties file to use something like this:

spa.spaPath=$SPA_PATH
spa.apiUrl=$API_URL

And then go back to leveraging envsubst? I don’t think they need to be actual values…

Well when I tried, the build failed. I think there is a subsitution happeing beforehand and it tells me the value is empty. Tried to escape it with \$API_URL but it also fail (I don’t recall the error). And then I stopped trying.

1 Like

So testing things out, it seems to work for me if it’s $SPA_PATH but ${SPA_PATH} causes an issue. I think that’s because the resources plugin uses ${...} to denote variables it should substitute.

I’ve pushed a update for the embedded Dockerfiles and a docker-compose-embedded.yml file that orchestrates them together.

I think we’re ready to start setting up a Bamboo job to build this branch and push the Docker images out there. Any objections?

Alright after a few more commits and messing around with things, we have a working CI pipeline to build 3.x and actual images for the gateway, the frontend, and the backend with all the necessary components embedded in them.

That build has two pathways, one to build a “dev” image with the latest ESMs (at the time it’s run) and one to build a “demo” image with the latest released versions of the ESMs (at the time it’s run).

Next, I think we need to get the necessary docker-compose stuff added to the ansible configurations and then get it deployed somewhere.

4 Likes

We’d really like to release 3.0.0 to the public within the next week if at all possible, and it would be ideal to do it on demo3.openmrs.org

Created this ticket for @ibacher to help us track this work: [MF-805] Set up demo3.openmrs.org (Demo/Production Environment) - OpenMRS Issues

(Sorry Ian, I know you have heaps on your plate, so this isn’t a complaint, just an effort to keep up our momentum)

demo3.openmrs.org is not a great subdomain, looks like there is a demo1 and demo2 and that demo3 is just there for redundancy.

That is great → o3.openmrs.org

(Amusingly O₃ happens to be the chemical element for trioxygen.)

1 Like

I actually really like o3.openmrs.org, so I vote we use that for the demo site. We’re also going to have a separate dev site. Do you have any naming suggestions (right now, it’s dev3.openmrs.org)?

1 Like

I’m alright with o3.openmrs.org. However, the original reason for demo3 was because of the current name, “demo.openmrs.org” for the 2.12 RefApp.

I think we need to plan ahead now, for the future where we’ll have removed the “3”.

So re. Dev Site - I am fine with dev3 for now; however, I think now is the right time to just leapfrog straight to “dev.openmrs.org”. (I see that this currently goes to the Add Ons site; not sure if anyone’s using that pathway though - I’ll see if I can find any data in our Analytics…)

@mksd @ibacher @dkayiwa are you okay with this approach? I realize some folks may be used to finding the 2.x experience at demo. and dev., but if we’re serious about our 3.x direction, then it’s probably good to explicitly head in that direction earlier. We can also make use of the Website and a Blog post to help make the community aware of the change. I can also kick off a dedicated Talk post before we switch over, and we can confirm with the TAC team on Monday too.

If it’s more expedient (due to the current 2.x living on demo.openmrs.org and dev.openmrs.org already going to Add Ons) I’m also fine with moving ahead with o3. and dev3.