Developing the "OCL for OpenMRS" Application

Development
1302

@mogoodrich I haven’t not came across that permission issue, also I have experimented with your dictionary and i’m getting same auth error

Also we at ampath are considering to start using the module immediately, I have been doing some testing to assess if the module is production ready, some of the things I fout out

  1. If you want to subscribe to multiple dictionaries say A and B, this is only possible if you do one at a time with specific dictionary subscription urls. I had read somewhere in the documentations that importing CIEL concepts will replace all existing concepts with the newly imported concepts that means some local concepts will be lost, however I experimented this by first importing concepts from dictionary A, I then added local concepts on top of A. I went ahead to importing from dictionary B.

After all that, all concepts from dictionary A and local concepts were retained and B’s concepts were added. That means implementations are responsible for managing their local concepts. Also With B’s concepts that already existed in A’s dictionary, concept duplicate exceptions were thrown, I would say this is the expected behaviour.

  1. Importing latest release of a dictionary reimports all the concepts that has already been importing from the previous import. This means if local changes were made to the concepts they will be lost

  2. Rolling back to a previous release is not supported, since dictionary versioning is not yet supported, I have tested this by removing some concepts from my current release and created a newer release, this did not remove these concepts from openmrs

  3. Does the module support existing implementations to start managing their dictionaries in OCL with some kind of one-time upload? . ie we at ampath we would wish to be able to upload all our concepts say in ‘Ampath-starter-template’ dictionary in ocl, be able to create another dictionary by copying the starter template and be able to add more concepts to it from CIEl from which we can then release and subscribe to. Also if this is not yet supported, I would like to request all of you to help brainstorm on a workaround for this

  4. Also we deem it fit that implementations should be able to roll back to their original concepts in case anything goes wrong during concepts import hence need to be able to backup concepts https://issues.openmrs.org/browse/OCLM-69

cc @mogoodrich @paynejd @dkayiwa @raff @ningosi @jdick

1303

@mogoodrich, at the moment exports can only be fetched by logged in users. You need to provide the authorization header in your API request. See https://github.com/OpenConceptLab/oclapi/wiki#authentication-and-authorization

Also have you provided your token when setting up subscription url in the module?

You can find your API token after logging in through a web UI at the bottom on the left side.

1304

Being able to subscribe to a single url was a design decision. The reason behind it was to encourage an implementation to create a single concept dictionary in the OCL service, which would include other dictionaries or their subsets. A tool, which should facilitate that can be tested at https://openmrs.demo.openconceptlab.org. We believe that it’s more correct to manage concepts in OCL and resolve any inconsistencies or duplicates there and only subscribe to a curated content from OpenMRS, because any conflicts are harder to resolve on the OpenMRS side.

This is not exactly accurate. If the previous import was without any errors, then the next update will only import content from the newer update. However, if the previous import had some errors, then we will re-attempt to import any previous updates which had errors together with the latest update. It is true, however, that local changes to the concepts may be lost when an update contains concepts, which have been modified locally.

This is true. We do not have a way to track removed concepts at the moment, but it’s also recommended not to remove concepts rather deprecate them, which can be propagated through an update.

It’s technically possible to have your OpenMRS concept dictionary uploaded to OCL. It’s a manual process, which we could possibly do for you. All what is needed is a dump of your MySQL database with concept dictionary related tables i.e. concept, concept_name, etc. Not sure, if we have any process in place to deal with such requests and I’m not sure what versions of OpenMRS are supported by our scripts so you would have to check with @paynejd for details on how to handle that.

No question about that! Do you have any development capacity to help with that? I’m available to conduct reviews or answer questions.

1305

Thank you for the clarifications, also i’ll liaise with you on jira about the backup ticket. I’ll follow up with @paynejd to have our dictionary uploaded to ocl

1306

@raff, thanks for your comments to @jecihjoy.

We will do our best to get this in a beta testing mode as soon as possible.

As @jecihjoy mentioned, it is critical for us to maintain a local backup of all concept related tables prior to any changes made to them by the subscription module. We need to ensure that we can locally restore our concept dictionary in the event that something unforeseen goes wrong. Given we may be among the first users of this service in the OpenMRS community, we need to be sure that any unanticipated bugs don’t cause irreversible changes. I realize this may be unlikely but we must be very cautious. @jecihjoy will make time to get this feature done asap with your guidance on the proper architecture.

I like the model you’ve mentioned above about having a single collection that an openmrs implementer would subscribe to. In our use case, we would like to create an AMPATH collection which pulls concepts from CIEL as well as the KenyaEMR collection (which we hope will also be created at some point on OCL). At this point, is it possible to import concepts from a collection (which are not in CIEL)? Or, is it only possible to add concepts from CIEL to a collection? This would be a big blocker for us if a collection could only import CIEL concepts.

Lastly, we would greatly appreciate it if you and @paynejd could assist with uploading our current AMPATH dictionary. We’d like to get this up asap so we can begin testing things out and mapping out our workflows. This initial import will serve more as a testing collection as we will continue to manage our dictionary locally until we feel the OCL/AMPATH setup is ready for production. Once we are close to that point, we would ask that you reimport our dictionary at that time. It’s important to us to have this “test production” set up and I hope you won’t mind assisting us twice with this process.

Thanks again for all your help.

JJ

1307

@jdick @jecihjoy sounds like PIH and AMPATH may have similar use cases. I wrote a message to @paynejd last week but realize I probably should have posted it here. I am about to board a flight to Maputo but will post it here quickly, it describes the main questions I had have a quick review of OCL, as well as some of our potential use cases. fyi @raff

  • What is a “source” is and what a “collection” is and how they interact? When I create a new “dictionary” in OCL for OpenMRS, it seems to result in both a source and a collection being added (see attached screenshots). Is this correct?

  • I know that OCL for OpenMRS only allows for single “preferred” source… what does it mean for a source to be “preferred”?

  • Is there support (in OCL and/or OCL for OpenMRS) for multiple users to collaborate on a source and/or collection?

Here’s my thoughts on how PIH might use OCL, and I’m wondering what your thoughts our on what OCL can and can’t currently support in this workflow:

We start by importing the current PIH concept dictionary into OCL. I would assume the dictionary would be a “collection”… I think the assumption would be that it would also be a “source”, but not sure if that’s 100% necessary for the use cases I’ll describe below… interested in your thoughts.

The PIH “collection” would be exported from OCL and imported into all of the “PIH EMR” instances… (TBD would be whether we’d use the subscription model and/or the JSON or CSV exports and/or some other format).

Use case #1

We (likely the Boston team, but could be one of our site teams as well) is building out a new form and/or feature intended for potential use by all of our implementations. We need new concepts to do this. Some of the concepts we need are in CIEL and so we just add them to the PIH collection. Some of them are in CIEL but we need to make some tweaks to them, so we copy them from CIEL but make some changes to them. Some are not in CIEL, so we create new concepts in the PIH dictionary. We cut a new version of the PIH dictionary/collection and our implementations update to it at their leisure.

For the concepts we created in the PIH dictionary that we believe are “CIEL-worthy” we issue a “pull request” against the CIEL dictionary, and, after some back and forth, come up with a final agreed-upon design. When that happens and the concepts gets adding to CIEL, we swap out our custom concept with the CIEL concept (maybe not even a ‘swap out’, but rather that the concept picks up CIEL mappings?). When the next release happens and the implementations pull in the latest release, everything goes flawlessly…

Use case #2

A particular country or implementation is building a form and wants to crank it out quickly. They add the new concepts for the form to a separate collection, say “PIH Haiti” or “PIH Mexico”. This new collection is then installed on top of the “PIH” collection on that particular country’s implementations.Similar to the process above, at a later point one or more of the added concepts may be proposed for inclusion in the PIH dictionary or possibly the CIEL dictionary. In that case, there’s as path for “moving” the concept from the “PIH Haiti” collection to the main “PIH” collection, and also adding it to CIEL.

Use case #3

Each country needs to maintain their own diagnosis set. So, within their specific “PIH Haiti” or “PIH Mexico” collections they would have a “Haiti Diagnosis Set” or “Mexico Diagnosis Set” concept. The concepts in the set would likely be a combination of CIEL, PIH, and country-specific concepts. When the an implementation wants to upgrade that list, they could just modify the list and cut a new release of the “PIH Haiti” collection and install it on their server.

Sorry, that ended being a bit of a brain dump. I certainly don’t expect OCL to support all this currently, but interested in your thoughts not just of how much effort it might be to add but also if you think I’m thinking about things the correct way.

Thanks for all your help with this! Really looking forward to how OCL could potentially improve our concept management.

1308

A “source” is where you define concepts. A “collection” is a group of references to concepts (and mappings).

In OCL for OpenMRS a Dictionary Xyz is represented behind the scenes by a source (think of this as “Custom concepts for Dictionary Xyz”) and a collection (which is the actual Dictionary).

The idea is that when you’re assembling your dictionary, you primarily pull from one or a few “preferred” sources. E.g. in PIH’s case, the preferred sources for Haiti could be (1) CIEL, (2) PIH shared concepts.

Right now OCL for OpenMRS treats CIEL as the hardcoded preferred source for everyone. This is okay for some definition of MVP, but it’s a serious limitation for a use case like PIH’s.

OCL supports sources/collections that belong to an Organization. And all the members of the org can collaborate on the source/collection. In OCL for OpenMRS, when you create a dictionary you choose whether it belongs to your individual user, or to an org you belong to. (Managing user-org membership has to be done through OCL directly.)

About your PIH use cases…

The ideal way to get started with OCL (but this may be too much work and coordination to actually pull of), would be:

  1. analyze the entire current PIH dictionary and determine which of your concepts are (1) custom PIH concepts, vs (2) pure copies of CIEL concepts.
  2. upload all of (1) to a new source like “PIH custom concepts”
  3. create a collection to represent the current PIH dictionary, which includes references to all concepts in “PIH custom concepts” and also references to (2) CIEL concepts you use.

The simpler/safer way is to just upload all of your current concepts as a new “PIH common” source, and also make a collection out of this.

(Either way, you should do some manual tweaking to make sure the source and collection are consistent with the way OCL for OpenMRS does bookkeeping.)

I’ll let @paynejd comment in more detail, but I’ll note that (a) the “pull request” flow is not implemented yet, and (b) “I create a concept in my own dictionary and start using it; later it gets pulled into an upstream dictionary, and the tooling supports this” is not implemented either.

1309

We can discuss in Maputo. but agree with Darius. I think we should be able to tell pretty quickly which PIH concepts are already in CIEL, although we would want only those that are identical to be sourced directly from CIEL in a collection.

1310

Thanks @darius!

I had assumed that we don’t yet have functionality to merge into an upstream dictionary, but is there functionality and/or a proposed workflow for handling the case where, say, we could allow different our different implementations to create their own concepts and then create a collection that is a combination of the PIH collection + their custom collections. One made use case we want to solve is to allow implementations to quickly add and deploy their own customizations when building forms.

Take care, Mark

1311

The idea would be that your implementation creates a dictionary that has two preferred sources: CIEL, and PIH-shared. (This is not supported today though.)

You could also start the implementation dictionary as a copy of a PIH starter dictionary. (May require some manual scripting today.)

We didn’t design the feature of having a dictionary/collection as a preferred source (i.e. if you want your central collection to be the top recommendations in the UI when building the implementation dictionary, rather than making a copy). That feature could be added I guess, with dev time.

1312

I am still unclear what the ACTUAL status of the OCL for OpenMRS code is. Has anyone been able to use the production site with the subscription module to populate an OpenMRS concept dictionary? @dkayiwa, have you?

1313

8 posts were split to a new topic: OCL for OpenMRS: Sources and Collections