We are making progress on being able to run openmrs-core with multiple replicas.
We need to make adjustments to the initializer module so that it only import files on the first node it runs and it doesn’t re-import on consecutive nodes. We also need to consider upgrading a node and having new import files. In such a case the first node that gets upgraded should run the import and other upgraded nodes should not.
I think that the easiest way to accomplish that is by persisting the import state in DB and not on disk so it’s visible to all nodes.
This seems like the more challenging thing to solve. Iniz basically figures out whether or not a file has been processed by comparing its checksum to the stored checksum. If two nodes for the same instance have different Iniz configurations, then nodes run after the “upgraded“ node will try to process their config, potentially over-writing the newer configuration. I’m not sure how to communicate “these checksums mean you do nothing“ without completely redesigning how Iniz checks for changes.
I think that kind of points to us needing a solution along the lines @dkayiwa mentioned. Even if we store the checksums in the database, we’ll still need to make sure that every node sees a consistent initializer configuration.
What about introducing a centralized database flag or table that tracks whether the import has been completed for a given version or config set.
The first upgraded node checks the database,if the import hasnot run, it proceeds and marks it as done and all the subsequent nodes also check this flag and skip the import if it’s already marked as completed.So we would avoid multiple nodes to import at once without needing to redesign how checksums are calculated or interpreted.
There’s no concept of a “version“ to leverage here though. So such a scheme is possible, but requires re-working Iniz in a way that might make Iniz more annoying. Currently Iniz scans through each file in it’s configuration and basically determines “has this file been processed“. Theoretically, I guess, we could keep track of multiple checksums for each file in the DB, but then we lose the ability to easily “roll-back“ the Iniz configuration.
Alternatively, we could mark just one node to run initialization via a system property… We would have to make sure that any upgrade begins from that node…
Yeah. It might be nice to have a feature where we could do this with “initialization“ modules in general. Most implementations end up with some sort of module that’s responsible for loading metadata and sometimes certain data elements, and they aren’t all based on Iniz, e.g., PIH’s pih-core or KenyaEMR’s CoreContext.
All right so I decided to go with having one instance as primary and setting a system property for modules to check and run any imports only on primary. It still requires making changes in modules, but it makes things much simpler.
Thinking again on this… I feel we need to look more towards the future, where the OpenMRS backend service is stateless and we don’t need to attach volumes to containers in Kubernetes thanks to using StorageService for handling files. No volumes means ease of replication and backup with StorageService and ease of moving replicas between nodes and handling failover.
I’d like us to re-consider saving checksums for imported files in DB and adding a manifest to a set of import files with a version. We would only run an import for files, if a version increases. We could also provide a way to roll back to a previous version with an environment variable, if an import to a newer version fails. It would allow us to stop relying on local disk storage and roll out upgrades using any node in a cluster with automated roll back. It would also speed up startup time as we would not have to iterate through all the files in different modules and evaluate checksums on each startup. Please see TRUNK-6418 and TRUNK-6417, where we are going with startup time improvements.
It’s slightly more work, but it brings us closer to the future, where OpenMRS backend service becomes stateless and all data is persisted via DB and StorageService (S3/Minio/local storage).
I would be supportive of re-considering moving initializer checksums to the DB. In addition to the use case presented here, we run into issues occasionally when restoring a database from a backup where initializer loads all of the configuration files on first startup because the record of which initializer checksums have already been applied to the database are not stored in the database itself. One needs to know this and have a process for both backing up and restoring the initializer_checksums directory at the time of DB backup and restore to prevent this from happening. It isn’t a huge issue, as Initializer loading should not generally change any state to an undesired value, but it can be an annoyance / factor if there are downstream processes that listen for DB change events, like sync, event firing, or a CDC mechanism…
I don’t have any issue with saving the checksum in the DB; that seems perfectly fine and, realistically, a lot more efficient use of storage. I think we would want to preserve one feature of the checksums I alluded to above, specifically, that there is only ever 1 checksum per file.
I’m ok with some kind of version manifest, but we probably need to figure out how that works, especially with features like content packages, and specifically, a distributions ability to contain many of these. Currently, we haven’t been requiring a single versioned artifact containing all metadata.
@mksd and @mksrom, since you own the project. Would you approve such changes? In summary:
Save checksums in DB, which solves the issue of having to backup checksums stored in a file system and replicate them between all nodes.
Introduce a version for an import set. If the version is higher than persisted in the DB examine checksums and do import. If version is lower, skip the import unless there’s an environment variable set to roll back to older version. It speeds up startup and makes it easy to do rolling upgrades without having a dedicated node to run an upgrade.