GSOC 2026: Support for horizontal scaling of OpenMRS instances.

Community GSoC
1

Hi everyone

I’m Suubi Joshua from Kampala Uganda, this summer I am working on Support for horizontal scaling of OpenMRS instances as part of GSOC 2026. I will be sharing updates about this project every week to show my progress and share any blockers, or for any one who might have any questions or clarifications on what I am doing throughout the project.

What is this project about?

Historically, many OpenMRS implementations have relied on traditional, monolithic single-server setups. While this works well for individual clinics, massive national-scale health networks or regional hospital systems can quickly push a single app server to its limits, creating performance bottlenecks and risky single points of failure.

This project aims to completely blueprint and build a production-ready, highly available, and cloud-native scaling architecture for OpenMRS on Kubernetes. By moving away from rigid, manual configuration patterns and leveraging modern infrastructure orchestration, the goal is to ensure OpenMRS can handle massive spikes in clinical traffic while keeping patient data entirely self-healing and continuously online.

How it works

To run OpenMRS across multiple servers simultaneously, I am breaking down the traditional “all-in-one” model into independent, coordinated layers;

  • Smart Startup: Automatically coordinates servers during simultaneous boots so they don’t fight over configuration files. The first server handles the one-time setup while the others wait their turn to start smoothly.

  • Shared Sessions: Syncs active user logins across the entire server pool so clinicians never get suddenly logged out if the system shifts their traffic to a different replica.

  • Unified Storage: Connects servers to a shared cloud storage network so patient documents or lab charts uploaded to one replica are instantly accessible to all others.

  • Clustered Database & Search: Moves from single database and search instances to cooperative clusters. This automatically balances heavy clinical traffic and keeps searches lightning-fast.

  • Advanced Routing: Isolates traffic management from the core application code, allowing deployments to seamlessly route clinical traffic and safely test out new software updates on a small subset of servers.

Key Goals

  1. Dynamic Scaling: Establish automated rules that seamlessly spin up new OpenMRS server replicas during peak clinical hours and spin them down when traffic drops.

  2. Zero-Downtime Upgrades: Fully transition our database and search layers to automated management systems capable of self-healing and running maintenance without going offline.

  3. Seamless User Experience: Ensure absolute data consistency across all server replicas so clinicians experience zero session drops or missing file attachments.

  4. End-to-End Security: Enforce mandatory data encryption for all internal communications between servers by default.

Resources

Thank you for you support so far.

cc: @raff @beryl @jayasanka

3 Likes
GSoC 2026: Weekly Blog Posts
2

Community Bonding Update

Hi everyone!

Now that the official coding phase kicked off, I want to share a recap of what has been happening behind the scenes during the Community Bonding period.

My mentor @raff and I used this time to align on the technical roadmap and architecture. We dove straight into the code to clear out legacy infrastructure blockers and lay down the foundational architectural pieces so that true horizontal scaling can happen smoothly.

Here is a breakdown of the milestones and pull requests (PRs) we have already successfully achieved and merged across our primary project repositories:

:hammer_and_wrench: What’s Been Done So Far

  • Mentor Alignment & Architectural Reviews: Held productive sync session to align on our implementation strategies, review the overall scope, and establish a clear development pipeline for the weeks ahead.

  • Built and Published Custom Elasticsearch Image (openmrs-contrib-elasticsearch PR #1): To support cluster-wide search functionalities natively, I built a custom Dockerfile that pre-installs the analysis-phonetic plugin. I also wired up a GitHub Actions workflow to automatically publish this image to Docker Hub. This has already been successfully reviewed and merged! Credit to @ibacher for the guidance on this.

  • Eliminated Startup Crash-Loops (openmrs-contrib-cluster PR #10): I resolved a critical race condition where application servers would crash-loop if they tried to boot up before the background search index cluster was fully ready. By introducing smart readiness checks, the servers now wait patiently, paving the way for safe, automated scaling (HPA).

  • Upgraded to the Official MariaDB Operator (openmrs-contrib-cluster PR #9): We completely ripped out legacy database blueprints and replaced them with a cutting-edge MariaDB Galera cluster. Database scaling is now entirely automated; the system natively supports synchronous data replication across multiple active database nodes.

  • Migrated to the Official Elastic Operator (openmrs-contrib-cluster PR #8): To ensure search operations scale smoothly, we moved search management over to the official enterprise operator, migrating from legacy Bitnami blueprints. This upgrade enforces secure, encrypted communications by default and allows implementations to split up heavy search workloads onto dedicated servers.

:rocket: What’s in the Pipeline (Current Work)

  • Modern Routing via Traefik & Gateway API (openmrs-contrib-cluster PR #12 - Open): Right now, I am actively working on replacing the old network routing setup with the modern Kubernetes Gateway API powered by Traefik v3. This will cleanly isolate traffic management from our application code, unlocking capabilities for safer software updates (like Canary or Blue/Green rollouts).

  • Re-architecting Shared Storage & Migrating from MinIO: I am currently evaluating a critical transition for our S3 object storage layer. Because the open-source community track for MinIO has been frozen and permanently archived, it introduces long-term security debt and unpatched vulnerabilities. To solve this, I am actively exploring and gathering community feedback on replacing it with a secure, highly active, and lightweight alternative, specifically SeaweedFS or Rook/Ceph to serve as our new unified, S3-compatible cloud storage engine.

:construction: Blockers & Next Steps

Fortunately, there are no blockers! The community bonding period has been incredible for aligning closely with my mentors and landing these prerequisite building blocks.

As we transition into the core coding phase, my immediate next steps are:

  1. Finalizing and merging the modern routing updates.

  2. Resolving the S3 object storage migration based on community and mentor feedback.

  3. Beginning formal load-testing to calibrate automated scaling rules for the core application servers.

A huge thank you to my mentor @raff for the stellar guidance, technical alignment, and rapid code reviews so far.

Medium Blog

6 Likes
3

Update

Hi everyone!

Here is a recap of what went down in Week Two of the official coding period.

This was a lighter week by commit count but a heavier one by decision quality, two major items moved forward: a significant routing upgrade got merged, and our long-running storage debate reached a community consensus.

What Shipped

Migrated Edge Routing from Nginx Ingress to Traefik Gateway API (openmrs-contrib-cluster PR #12 — Merged Replaced the legacy ingress-nginx controller with the Kubernetes Gateway API powered by Traefik v3.x. Key highlights:

  • Introduced a new helm/traefik-gateway/ chart as a cluster-wide, reusable gateway
  • Replaced all kind: Ingress resources with native HTTPRoute resources in both the backend and frontend charts
  • Preserved legacy Ingress templates (disabled by default) for compatibility with K8s providers that still rely on classic ingress
  • Solved a multi-replica login failure caused by CSRFGuard token mismatch across pods, implemented cookie-based sticky sessions via Gateway API + TraefikService, with automatic activation when replicaCount > 1

In Review

Replace archived Bitnami MinIO with SeaweedFS (openmrs-contrib-cluster PR #13 — Open): Following the community discussion on Replacing MinIO with SeaweedFS or Rook/Ceph, and with clear direction from @raff and community member @ranidunethma in favour of SeaweedFS for its simplicity and lightweight developer experience, this PR replaces the unmaintained Bitnami MinIO subchart with the official SeaweedFS chart. Currently addressing one review comment from @raff on whether the filer component is needed in our S3-only setup.

Next Steps

  • Land PR #13 (SeaweedFS)
  • Begin the openmrs-operator Helm chart, the automation layer that bootstraps all prerequisites so a new cluster deployment is a single helm install away
  • Begin application-level readiness coordination to ensure the OpenMRS app layer waits correctly for all dependent services

Medium blog.

4 Likes
4

Project Updates

Hi everyone! Here are some updates on the status of project.

One-command local deployment (openmrs-contrib-cluster PR #14): Consolidated the entire local Kind development flow into a single make deploy. New helm/openmrs-operator/ umbrella chart bundling MariaDB operator, ECK, and Traefik into one install; a bootstrap.sh automating cluster creation, local registry setup, and app deploy; a local Docker registry (kind-registry:5000) replacing the slow kind load docker-image workflow; cleanup of dead code along the way.

SeaweedFS replaces MinIO (openmrs-contrib-cluster PR #13 — Merged): Closing out the filer question I’d flagged in Week Two, OpenMRS clusters now run on an actively maintained, S3-compatible storage backend instead of a frozen one with an unpatched CVE.

Two openmrs-core contributions:

  • PR #6193 — adds a credential-verified database readiness check to startup.sh, fixing the same InnoDB-recovery timing gap found in PR #14, but at the application layer. Got a great catch from @dkayiwa in review: the fix as first written would have broken the dev image (no MariaDB client installed) and PostgreSQL deployments (the check was hardcoded to MariaDB). Both fixed.
  • PR #6210 — a small CI fix, split out from the PR above, that stops the OWASP dependency-check workflow from failing on every single fork PR due to an empty NVD_API_KEY secret.

Subtask planning for multi-tenant deployment: Broke down TRUNK-6483 (deploying backend/frontend only for an additional tenant, sharing the rest of the cluster’s infrastructure) into nine reviewed subtasks, now marked ready for work. One subtask shifted direction mid-review: instead of a Terraform module for tenant deployment, we’re now leaning toward an OpenMRSTenant custom resource built into the operator chart, so creating a tenant becomes “apply a CRD instance and let the operator handle the rest.”

Next Steps

  • Land PR #14 (make deploy) and PR #6193 (startup.sh DB-auth check)
  • Merge PR #6210 (CI fix)
  • Start TRUNK-6652 — the skeleton openmrs-tenant Helm chart
  • Work out the OpenMRSTenant CRD design

Thanks as always to my mentor @raff for the continued guidance.

1 Like
5

Week 5 Coding Update

Here is a recap of Week Five. Nothing here was new work so much as it was finishing work started in previous weeks, properly, all the way down to the lockfiles, plus a fourth thread (multi-tenancy) starting its first real review cycle.

:white_check_mark: What Shipped

The last of MinIO, gone for good (openmrs-contrib-cluster PR #17 & PR #18 — Merged): PR #13 (Week 4) migrated all the templates to SeaweedFS, but Chart.yaml still declared minio as a dependency, which made helm dependency update fail trying to fetch the archived Bitnami chart. Split into two focused PRs on review feedback (one change, one PR): #17 removed the dependency itself, #18 swept the remaining references out of the README, CI workflow, and Chart.lock. Worth noting: Chart.lock is machine-generated, so fixing it meant regenerating it fresh rather than hand-editing — MinIO is now gone from every file in the repo, not just from runtime.

DB-auth check lands in openmrs-core (PR #6193 — Merged ): This is the real fix for the Galera pod-hang bug. The original workaround was a Helm-level init container, but the better call was to fix the readiness check at the source in startup.sh itself, a chart-level workaround is invisible to anyone deploying OpenMRS outside our charts. Review caught two real issues before merge: the dev image had no MariaDB client (would have broken every docker-compose dev setup), and the check was hardcoded to MariaDB (would have broken PostgreSQL deployments). Both fixed. A small CI fix that had been riding along split out into its own PR (#6210, merged Jun 24).

Backporting to where the fix actually ships (PR #6225 → 2.8.x, PR #6226 → 2.9.x): Release images build from maintenance branches, not master, so the fix needed to follow. Opened both backports the same day #6193 merged, straightforward cherry-picks of the same commit, with minor context differences between branches. Both pending first review.

:counterclockwise_arrows_button: At the Finish Line

make deploy (openmrs-contrib-cluster PR #14): Approved this week after the init container was dropped (superseded by PR #6193) and a mariadb-operator bump to 26.6.0 resolved a Galera DDL deadlock. One round of local testing is the last gate before merge. This replaces what used to be 10+ manual steps across four README sections with a single idempotent command.

Tenant chart skeleton (PR #19, TRUNK-6652): First subtask out of the TRUNK-6483 multi-tenant breakdown. Deploys just a backend and frontend for an additional tenant, sharing the cluster’s existing MariaDB, Elasticsearch, and SeaweedFS rather than bundling dedicated infra per tenant. Deliberately incomplete at this stage, no routing, manual DB setup required — with TRUNK-6653 (automated DB bootstrapping) as the next piece.

:rocket: Next Steps

  • Get PR #14 over the line — local test is the last gate before merge
  • Begin TRUNK-6653 (Helm pre-install Job for tenant DB bootstrapping) and continue through TRUNK-6483
  • Get the open PRs merged
  • Keep an eye out for further loose ends now that MinIO, the DB-readiness fix, and make-deploy are all converging

Full write-up on Medium

Thanks as always to my mentor and the OpenMRS community!

4 Likes
6

Well-done @suubi7 , thank you for the great work :handshake:

1 Like
7

Hi everyone! It’s been a couple of weeks since my last update here. Here is a recap of what happened across Weeks Six and Seven, plus a midterm evaluation video.

:clapper: Midterm Evaluation Video: https://youtu.be/NvsHO_IgRYQ?si=0xggkz1gV8KYwq0M

:white_check_mark: What Shipped

Dashboards exposed via Gateway API (openmrs-contrib-cluster PR #20 — Merged): Operational dashboards are now accessible out of the box at localhost:8080 with zero DNS configuration:

  • Grafana at /grafana/ — logs dashboard backed by Loki
  • SeaweedFS Admin at /seaweedfs-admin/ — unified cluster overview and file browser
  • Root redirect — localhost:8080/ → /openmrs/spa/home/

One-command local deployment (openmrs-contrib-cluster PR #14 — Merged): Collapsed the entire local Kind development flow into a single make deploy. A new openmrs-operator umbrella chart bundles the MariaDB operator, ECK, and Traefik into one install. A local Docker registry replaces the slow kind load docker-image workflow. What used to be 10+ manual steps across four README sections is now one idempotent command.

SeaweedFS fully replaces MinIO (PR #13, PR #17, PR #18 — All Merged): The unmaintained Bitnami MinIO subchart is completely gone — templates, dependencies, lock files, CI references, and README. Clusters now run on actively maintained SeaweedFS.

:beetle: Release Preparation & the CrashLoopBackOff Investigation

With PR #20 merged, I started preparing the first major release: updating Helm charts to use the new 3.7.x-no-demo backend and 3.7.x frontend images, bumping chart versions, running helm lint, and deploying to a fresh Kind cluster.

Everything looked clean until make deploy — the backend pod entered CrashLoopBackOff, failing the DB-auth check in startup.sh with “Database did not accept credentials after 30 attempts.” The database was reachable and the credentials were correct.

Root cause: PR #6193 added a mariadb -h ... -e "SELECT 1" check to startup.sh. On master (Debian-based), apt-get install mariadb-client provides MariaDB 10.11, which ships the mariadb binary. But the ref app distro image builds from openmrs-core:2.8.x, which is Amazon Linux 2 — where yum install mariadb gives MariaDB 5.5, which only ships /usr/bin/mysql, not /usr/bin/mariadb. The mariadb binary name wasn’t introduced until MariaDB 10.5+ (MDEV-21303).

Verified empirically: mariadb returns exit code 127 (command not found) on the nightly ref app image; mysql with identical flags returns exit code 0.

My proposed fix: One word change in startup.sh — mariadb → mysql. The mysql client binary exists on both AL2 (MariaDB 5.5) and Debian (MariaDB 10.11). No Dockerfile changes needed. The original commit was reverted by @ibacher to unblock the ref app release, and the proposed fix is being discussed on TRUNK-6650.

This check matters for K8s deployments specifically because TCP port 3306 accepts connections before InnoDB/Galera is ready to accept queries — roughly a 12-second gap — which causes Liquibase deadlocks on fresh cluster deployments.

:building_construction: Tenant Chart Iteration

Tenant chart (PR #19 — Under Review): Three commits this week addressing gaps found during testing:

  • emptyDir fallback when no storage class is available
  • Configurable API URL and storage type per tenant
  • Missing frontend ServiceAccount added; RBAC resources gated properly
  • Infinispan gate removed from Role/RoleBinding to match the parent chart

:rocket: Next Steps

  • Hopefully land the upstream fix — mysql binary name change into startup.sh on 2.8.x and 2.9.x, rebuild core images, rebuild the ref app image, and cut the release
  • Continue tenant chart review — address remaining feedback on PR #19
  • Scope TRUNK-6653 — Helm pre-install Job for automated tenant database and user creation

Full write-ups on Medium.

Thanks as always to my mentor @raff and the OpenMRS community!

2 Likes
8

Major Updates

:building_construction: Made the core charts reusable (TRUNK-6736)

The biggest change is structural. openmrs-backend used to bundle its own infrastructure (MariaDB CR, SeaweedFS, Grafana/Loki/Alloy, Elasticsearch) as embedded dependencies and so anything reusing the backend also dragged in a full copy of the primary’s infra.

openmrs-backend and openmrs-frontend are now infra-free, workload-only charts; all infrastructure moved up into the openmrs umbrella, which deploys it directly. The backend now renders only its own resources (StatefulSet, Service, ConfigMap, Secret, routes). Since this was breaking, the charts moved to 2.0.0 with a fail-fast guard that rejects old 1.x value files instead of silently dropping resources.

:bug: Fixed silent cache clustering (TRUNK-6671)

Infinispan clustering was forming singleton caches and each pod thought it was alone because JGroups DNS_PING resolved the ClusterIP service (a single VIP) instead of individual pod IPs. Fixed with a dedicated headless Service (publishNotReadyAddresses: true) so DNS returns every pod, and removed the Infinispan ports from the ClusterIP service (an unintended exposure).

:rocket: Multi-tenant deployment, rebuilt on the shared charts (TRUNK-6652)

With the shared charts reusable, I rebuilt the tenant chart. Instead of forking the backend/frontend templates, openmrs-tenant is now a thin umbrella that depends on openmrs-backend/openmrs-frontend and supplies only per-tenant config — DB connection, SPA settings, labels. No duplicated templates, so tenants inherit every reviewed shared-chart fix with zero drift. This is still open and under review.

:package: Cut a major release (2.0.0)

Published 2.0.0 of the openmrs, openmrs-backend, and openmrs-frontend charts to the Helm repo. Major bump because the refactors breaks any 1.x values file. Also bumped the app images to OpenMRS 3.7.x no demo data for faster setup.

Thanks to @raff my mentor and @dkayiwa for the reviews and guidance.

3 Likes
9

Nice work! How will you tell the front end what backend to use?

10

Hi @shawnjots, frontend and backend share a namespace. The frontend uses a relative API_URL: “/openmrs”, and Traefik (via Gateway API HTTPRoutes) path-routes /openmrs/* to the backend and /openmrs/spa/* to the frontend with a URLRewrite that strips the prefix. No backend discovery needed.

For multi-tenant the plan is for each tenant gets a per-tenant hostname (coast.openmrs.example.com, etc.) on its own HTTPRoutes. Traefik selects the tenant from the hostname before path routing kicks in, so the frontend still never needs to know a backend URL.

1 Like