Spent some time brainstorming about this and thinking about your points from yesterday @bistenes… makes me realize I probably need to spend an extended amount of time to really think things through and pull my thoughts together, but rather than taking longer than I’d like to get that done, here’s a laundry list of thoughts:
In your point that starts “I think the biggest technical hurdle in proliferating…” I don’t think that the Hibernate DAO’s really have anything to do with it? I’m thinking there’d likely be a new bean or beans created that would be wired into modules. Each module would have to be updated manually to use this new bean. Beyond the technical issue of trying to override things outside the module, each module tends to do configuration in a different way, part of this project would be about reworking each module to use a standard config format… tedious, but I don’t see a way around it.
There’s definitely metadata, like encounter types, visit types, concepts, etc, that will still need to be persisted in the database and that modules would still access via the existing services, and this new architecture would simply be responsible for making sure the appropriate types installed in the DB and/or configured properly
You mention using the SDK in a few places… currently at PIH we use the SDK only used to set up a development environment; we don’t use it at all to set up production servers. I believe it does have some features related to production, but I haven’t explored them (just glancing at them now, I’m interested in the docker features @mseaton). Definitely something to consider (the SDK has become a great tool) and I’d support expanding it, but just wanted to clarify since you mentioned the SDK in the context of a production deploy.
In perhaps a “file-under-unhelpful” comment, your different config tree examples makes me realize that there are many ways we could config things and nothing strikes me yet as a “best” option… yet it would be easy to come up with one that we look back on as “bad”
I’m still not in love with the term “verticals”, though if others like it and there’s no better suggestion, I’m fine with it. What we are really talking about it configuration “bundles”, right? Vertical seems too tied to actual clinical programs. For instance, I could see there being an “HIV” bundle and a “Haiti” bundle installed in the PIH EMR, but those two things are really perpendicular so they both can’t be verticals, right? Or am I getting bogged down in semantics?
It might make sense to distinguish between configuration and content? For instance, in terms of configuration, the EMR-API module needs to be told what encounter type to be used for check-in, which provider to be used a the “unknown provider”, etc (see EmrApiProperties–these all used to be handled via Global Properties, but in EMR-API they’ve been switched to use Metadata Mappings… this migration is a work-in-progress, but I think we seriously want to consider focusing on using Metadata Mapping going forward); In terms of content, an HIV bundle may provide a new HIV intake form and associated concepts and encounter type
For configuration, using the EMR-API example, for a fully seamless deploy, EMR-API would need to be bundled, say, with a default encounter type to use for Check-In, a default provider to use as Unknown Provider, etc, and then via configuration the implementer could tell EMR-API not to install that metadata but use existing metadata
For content, one big challenge is how to share a form with all the concepts it requires. Metadata Sharing and Html Form Entry actually provide a technical way to do this that works pretty well (full disclosure, I wrote it 9 years ago )… when you share a form via Metadata Sharing it bundles all the concepts it uses with it, but for various reasons this hasn’t been particularly useful for us–MDS packages are rather opaque as you have noted. Also, this can lead to real smorgsabord of concepts… ie, what if the HIV vertical and the TB vertical provide a different concept for “pregnant”? When importing a package you’d like to be able to either map to an existing concept in your dictionary or use the default one that the package/bundle/vertical provides. This is a bit of a nasty problem, and the solution probably revolves around some combination of trying to use a standard concept dictionary (CIEL) as much as possible, and using Concept Mappings (after I write this I realize that this is a similar problem as with configuration, but I still think there’s a potential distinction between the two)
Agree with your point about the “main thing”… we need to find a path that allows us to move forward iteratively (which my brain dump of thoughts here may not help in achieving). At a high level, I think your idea of starting with a new module makes sense, but interested in other’s thoughts. The new module could then use the Iniz module (or perhaps the Metadata Deploy module) as it’s method of installing metadata into the database.
Continuing my late replies with only partial context…
These are some far-reaching, interrelated, and delicate changes. So +100 to the idea of doing things incrementally.
Mark’s point that just the XML part of an HTML form is not necessarily enough, because the form itself requires metadata is important. This includes mainly concepts, but also encounter types, and perhaps other things. So a “vertical” may also need to package metadata along with config.
I also like “bundle” more than “vertical” as a name.
Mark already mentioned it, but the metadata mapping module took a stab at part of this, i.e. it allows the EMR API module to
+1 to distinguishing between content and config (even if they may be technically similar). Also +1 to looking at metadata mapping, which already does provide a way for the emrapi module to say “I need to be configured with an encounter type that’s used for check-ins”.
Okay, so there’s definitely some difficulty around config vs metadata (vs content? I’m a little confused about what the line between the content and metadata is supposed to be, @mogoodrich ).
I’ve been envisioning three separate mechanisms. The config loader loads configuration into memory. Iniz loads non-concept metadata into the database. And something else (eventually OCL) loads concepts into the database.
I had been thinking that users would continue to be responsible for ensuring that all the metadata that their config refers to is present, though all references to metadata would be validated by the config loader.
But actually, it seems like it should be quite feasible for bundles to contain both config and Iniz-style non-concept metadata. And it would seem reasonable for Iniz and config loader to work together to validate that the config and metadata provided work together.
Mekom also uses Iniz for Concepts. Were we not headed toward an OCL world, I might advocate going sort of in that direction, which would allow us to easily bundle concepts as well.
This raises the question: does OCL-for-OpenMRS intersect with this problem? How do we imagine bundling concepts in some way that cooperates with OCL?
I need to go learn more about MDS and Metadata Mapping both.
What I can say about Metadata Mapping at this point is that, though I’ve never worked with it, it seems like a great idea and something that the config loader and Iniz should know how to handle, or maybe even adopt as the official way to name and refer to metadata.
Somewhat as an aside, I recently found out about HCL, which seems simply better than YAML, and now advocate using that instead. Hopefully that doesn’t seem wrong to anyone.
Will follow up on the other points, but re: JSON vs YAML vs HCL, I’d err on the side of something that is more well-known and widely used, which seems to favor JSON, but, of course, that’s not the only factor. I haven’t heard of HCL before (which may not mean much )… do we know how widely adopted it is?
@bistenes you are right, metadata and content are probably more or less then same, and, if not, whether or not we call something “metadata” tangential to whether it’s config or content.
Content:
Location “Unknown Location”
Config:
Metadata Mapping (previously Global Property) that tells EMR API to use “Unknown Location” as the location when no location is specified
Content:
Visit Type “Clinic VIsit”
Config:
Metadata Mapping that tells EMR API to use “Clinic Visit” as the visit type when no visit type is specified
Content:
An HIV intake form with it’s associated concepts and encounter type
Config:
A default App and/or Extension wiring the HIV intake form as a link on the Visit Dashboard
On top of that (this distinction may not be important, but I throw it out there): the “Clinic Visit” and “Unknown Location” are both required content & config that OpenMRS needs to run, while the HIV content/config is optional and provides new functionality.
Another point about the HIV intake form… so this could be packaged an “HIV” bundle that provides as content the form, the concepts, and the encounter type, and the config that adds the HIV intake link to the “Visit Dashboard”, but an implementer may want to override/disable that app and provide their own config that makes it appear on, say, a custom “HIV Dashboard” they’ve set up instead. I think supporting this kind of override is mandatory… or if we don’t want to support that yet, we ship the bundles without any config and require implementer to do the wiring.
Getting into more complex features: it’s also possible that the implementer wants to override the encounter type and one or more of the concepts used on the Intake form. This could (should?) be accomplished by on the form having all concepts referenced by Concept Mappings (https://wiki.openmrs.org/display/docs/Concept+Terms+and+Mappings) and the Encounter Type referenced by a Metadata Mapping… the HIV bundle could provide it’s own Concept Source/Metadata Source (basically just a unique key/name)… then if an implementer wanted to use these own concepts and/or encounter types they could add their mappings to their existing concepts and configure the Bundle not to import those concepts when installing. Anyway, I don’t think this is something we need in “Phase 1” (besides perhaps setting a best practice of how we define/reference concepts and other metadata within forms) but wanted to throw it out their as a vision in case it affects design of “Phase 1”. Hope there’s some food for thought there…
The thing with OCL is that it sort of “transgresses” our packaging mechanism. We like the idea of packaging self-sufficient distributions and ship them wherever they have to be deployed. Having OpenMRS instances being fed directly from OCL means that the software package would not contain the necessary concepts to run the distro, or would not necessarily contain the latest set of concepts for that distro since this latest set is supposed to come from OCL. Also, and simply, back then OCL itself was not ready yet, let alone the OWA client.
@mksd@mksrom It’s been brought up that Iniz and Metadata Deploy do pretty similar things. Could either of you explain briefly what’s the difference between Iniz and MDD? Why doesn’t Iniz just use MDD, as seen under “Object sources” here?
I looked at MDD initially and it came across rather as an API. I think perhaps Iniz should depend on MDD and use it to install entities. If that’s the correct reading I’d be more than happy to support that transition rather than having Iniz (re)define utils all of all sorts (such as this one).
@bistenes Thank you for all the thought driving implementation, and thought I would add my thoughts from an implementor’s perspective. What are the main challenges when putting together an OpenMRS implementation and maintaining it:
Adding and updating metadata
concepts, encounter types, identifiers
Forms - depend on the above
Dashboards and apps - to display data from the forms
Reports these are usually java code (hopefully some configuration for the cohorts and datasets later)
Data format
I would go for JSON over YAML (better tooling and more consistent usage across OpenMRS already for apps) with CSV being a follow-on (probably leveraging an inbuilt editor to help manage this for non-technical usage)
Metadata storage - in the database but cached in local folders for use would speed up the performance. The local cache can always be manually refreshed (There is a rebuild search index feature that does this for concepts)
Ability to load metadata via version controlled files would be great (Iniz) plus being able to do this in a custom module (even better) since this is the approach that isolates dependencies to what is needed
I agree with @darius on using “bundles” rather than “verticals”
@mogoodrich on overriding forms from “bundles” what we have discovered is that if you have a form with the same uuid - then the latter will overwrite the deployed one in the bundle, which seems to be logical
It would be great if metadata modules were leveraged for this functionality to avoid having the same code in multiple places
Also would be great to be able to define a default config hierarchy that is controlled by a global property so that multiple distributions can leverage the same implementation, what PIH has is similar to what we are experiencing with UgandaEMR though we are using custom modules for that
All this would be great leveraging the SDK as the build tool which means there are too many scattered scripts etc
Just to address this comment @bistenes - the idea as I have understood it has been to enable OCL to feed concepts to an implementation in 2 ways:
Export a “bundle” of concepts from OCL (likely as JSON), add these directly to your distribution package (today that would typically be in the resources of a module, but in the future perhaps in some sort of config bundle), and then have a mechanism (MetadataDeploy, Iniz, ocl-module, etc) that is tasked with reading that JSON file at startup and installing the concepts into the dictionary.
Provide a means to pull in and install these same JSON resources into an implementation as needed (either via a subscription or via a manual upload process)
My thinking is that the team pulling together distribution versions would do #1, but that the same metadata packages that are slotted for inclusion in a later release could be make available for installation and usage at an earlier point in time, whenever they are versioned and released.
So, in this way I don’t think this is at all inconsistent with our packaging or versioning vision.
Thanks for the feedback Stephen. Just a few notes:
YAML and HCL are both supersets of JSON, so implementers would be welcome to write JSON configs if they prefer.
Yeah, this seems like a pertinent problem. I think ideally you should just package such a “default config hierarchy” as a bundle. But I guess you would probably want to have built this using some preexisting bundles, making this a bundle of bundles. Which introduces a dependency tree problem. I think there are two reasonable solutions:
git fork your config repos
Create an SDK command to “flatten” a config tree into a bundle, merging all the configs together
“flat” dependency management of bundles in the style of Ruby Bundler (i.e., exactly one version of any bundle can exist in a given tree)
Approach #1 is the thing that happens without writing any code, and I suspect it might be the best approach. But down the line I think writing support for (2) or (3) (my preference in that order) wouldn’t be unreasonable, at least to try.
Apologies for not getting my responses here earlier enough and probably not consuming all responses here well enough, i have had this post waiting my response for quite some time until i allocated it time today. since i missed the design forum, i could be outdated in my responses and so please bare with my outdated responses
Depends, if it’s one organisation supporting this, it’s practical to have all configs initialised in one module iniz in this case, with more organisations supporting OpenMRS and some even more interested in more custom configurations, it’s better if they are to be maintained & initialised in core so any module can provide what needs to be initialised straight away if they want use such an ini-built initialiser than have to update iniz to add their configs.
a non technical team adding metadata if it would be initialised from such a configs directory makes sense, but besides setting existing configurations, in all cases configurations are added by technical people who are writing functionality to consume them making it almost of no use adding new configs by users except if they will write some reporting/groovy or some other sort of scripting allowed in openmrs to consume their new configs.
If loading all iniz configs in memory lazily or which ever way is not gonna work practically and we will have to load things from the database, iniz only handling initialising stuff into the DB, then metadatadeploy is doing what iniz is trying to achieve. we probably rather than define java objects need to make metadatadeploy accept data files json/yaml or both.
i will read through and add my responses to dialogs after quote below as soon as i can.
