Migrating PIH dictionary to OCL

Yes, by my analysis there are 58 concepts that are missing numeric “PIH” mappings in that provided database snapshot. I’ve detailed this out in another message internally @ball . We can hash it out there.

Updates:

Since the last post on this thread there has been a lot of discussion here: PIH dictionary import · Issue #45 · OpenConceptLab/ocl_issues · GitHub

Update today from our OCL call:

  • It sounds like @ball and @burke have addressed the majority of the issues with the import tooling (there was also a kudos shout-out to @mseaton for all the work he’s been doing to help this whole process)
  • The next step is for @mseaton to do another pass with the subscription module. @mseaton/@mogoodrich does that sound about right?

Please let me know if there’s anything I can do to help :slight_smile: I’d love to succeed at making OCL tooling useable for you; the earlier in 2022 the better!

Yes, I have this on my radar @grace and will likely do this in the coming week.

1 Like

Update here @grace - testing went well and I was able to confirm that the concepts exported from OCL matched those from our existing system, with the exception of a few known areas that we can manage separately. Next steps for us are to step back and evaluate the next phases of work to allow us to fully proceed with OCL adoption. Our aspirational goal is to achieve OCL migration this quarter.

1 Like

Thank you @mseaton for that great update!! Also delighted to see your comment in GitHub on the same.

So, let me make sure I understand this: The next steps for achieving OCL migration this quarter are:

  1. Source handling (what’s the use case/issue here?)
  2. Internal PIH discussion on next steps to fully adopt OCL

@mseaton @ball is this about right?

The remaining steps I have to get OCL ready for PIH to manage their dictionary within OCL:

  • Concept sources
    • Including concept sources in export (#1115)
    • Defining explicit conventions for source name equivalency (e.g., case-insensitive, ignore whitespace and non-alphanumeric characters)
    • Creating a plan for adopting/favoring use of canonical URLs for sources across OpenMRS and OCL (e.g., how we associate http://loinc.org with LOINC source in OpenMRS & OCL and include it in import/export/API steps)
  • Including retire reason in OCL import (#1118)
  • Ability to retain/track voided (aka retired) concept names within OCL (not currently supported by OCL’s model)
  • Gap analysis of day to day concept management needs for @ball (i.e., what does she rely on being able to do in OpenMRS that cannot yet be done in OCL)

@mseaton and @ball may have others.

p.s. Given Mike’s comment, I finally closed OCL’s longstanding ticket for PIH dictionary import (#45). Yay! :partying_face:

Thanks @grace and @burke . Some high-level steps:

  1. As @burke describes, we need to fix things such that when we import from OCL, we don’t create duplicate concept sources. The various bullet points that Burke describes are on point, and in the short term we can likely just get away with creating the conventions for source name equivalency and then implementing that in the OCL subscription module. We can probably evolve this ticket here: [OCLOMRS-1075] Improve creation of concept sources if necessary - OpenMRS Issues. I believe that will likely unblock us here, though the broader support for including richer concept source data in exports is still valuable.

  2. The bullet points @burke lists about retire reason and voided names are not super important to us. If everything else were ready to go and/or if we decided we didn’t want to support those, these wouldn’t prevent us from moving forward. They are firmly “could” issues.

  3. We have some time carved out this month for @ball and others on our team to spend time with the OCL for OpenMRS interface and mirror our daily concept creating and management work using that UI, to gain more confidence that we can move onto this platform without a loss of important functionality. This will be ongoing, and @ball is best positioned to communicate when she feels confident that all of our various use cases have been met. A pre-requisite for this work is that the latest OCL for OpenMRS snapshot code (and the latest OCL backend that has changes we need) is available for us to use in testing with our imported dictionary in either staging or QA, whichever is more appropriate. Can you confirm this after our discussion earlier this week @burke and @grace ?

  4. We need the work to add OCL support to Initializer ([OCLOMRS-950] Enable use of Iniz for OCL-based imports - OpenMRS Issues and also ensure that this work covers the use cases I laid out in a recent Slack conversation (and that this approach is agreed as a valid use case) - that Iniz and the subscription module combination should support loading multiple OCL zip exports, and one should be able to configure the order in which these load. This latter point is not yet ticketed. See Slack conversation leading up to and responding to this: Slack

  5. Once this is in place, we need to think about the next phase of dictionary import testing. Namely, that we likely want to have multiple sources and collections in OCL - a “PIH” one, and one that represents each of our implementation’s specific concepts that are managed outside of the PIH source. And then we need to provide imports into these different sources.

  6. Along with (5), I’ve noticed that when we upload our dictionary to OCL, all of our concepts are loaded in the PIH source, even if these concepts are shared CIEL concepts. I’m curious about whether this is correct, or whether we should try to identify - if a concept that we are uploading is unchanged from what is in CIEL, and if so then add it as a reference rather than as a new concept. But I’m also curious about how that then evolves later on - i.e. if we want to add custom mappings to it, or modify/add some of the names, is it easy to convert this “reference” to a new concept in our source that points back to the original concept by shared mappings? I’d like to learn more about this.

  7. We then need to spend time actively testing the process of removing MDS packages from our configuration projects and replacing them with OCL zip files, and confirming that both new installations and updates to existing installations work correctly, and that further updates to these packages work correctly. We haven’t quite worked out the details of this testing. One complication is that our dictionary in OCL Staging is out of date almost immediately after we provide it to @burke for upload, and so there is never an apples-to-apples comparison possible between the concepts installed by our latest distro and what is available in OCL to test with. It would really help if we had a way to automatically keep OCL staging up-to-date with our latest dictionary. @burke - thoughts?

There is likely stuff I’m missing, but these are some of the main items on my radar.

Mike

@ball and @mogoodrich FYI

1 Like

For now, i.e., without the ability to customise concepts, this seems like it’s the correct thing to do. Otherwise, if we converted those concepts directly to CIEL concepts there would currently be no way to customise them at all. Once concept customisation becomes a thing, it might be worth rethinking this.

There are some other considerations here:

CIEL uses a fixed-uuid scheme for the component parts of concepts (concept_id is padded with AAA…, concept_name_id is padded with BBB…, concept_description_id is padded with CCC…, concept_mapping_id seems to be padded with ABB…). The PIH dictionary doesn’t seem to follow the same UUID schema so that if those UUIDs are referenced anywhere, it might result in data or code referring to a concept UUID that no longer exists.

Similarly, CIEL has some concepts that map to PIH concepts, but all PIH concepts now (in OCL) have a SAME-AS mapping to a PIH “gold” code. Most of those mappings would be lost.

Not easily. The problem isn’t really creating a concept from an existing concept or adding a mapping to it to point to the original concept; it’s all to do with the other mappings (SAME-AS, but also Q-AND-A, and CONCEPT-SET). First, when creating a “clone” of a concept, we need to create new mappings from that concept to all of the concepts the original concept was mapped to, then ensure than any concepts they are mapped to are also correct. (This is where the cloning feature is stuck). Second, since this would be a change to an existing concept, effort would have to be made to update all the mappings that point to that concept to point to the newly created concept. Neither of those are impossible; it’s just not easy.

This touches on one of the key data model differences between OpenMRS’s concept dictionary and OCL. In OpenMRS, a mapping to a concept is just some plain text that external business logic can resolve. OCL has two kinds of mappings: external mappings which are similar to OpenMRS-style mappings (i.e., they are basically plain-text and not real links) and internal mappings, which are an actual link between two OCL concepts potentially in two different sources.

This is part of the reason why concept customisation is more of our long-term strategy here. A draft of some of how we’ve been talking about doing this can be seen in this very long Google Doc.

The quick hack here would be to give the zip files ordered names, e.g. 001_base_concepts.zip 002_tb_program.zip, etc.

https://openmrs.qa.openconceptlab.org is once again pointing to the OCL staging server, so you should have the latest Dictionary Manager and the PIH source available there.

In February and before making the switch, I expect to do a weeklong side-by-side testing of our current antiquated concept management with using the OCL concept manager. I wish I had a better test plan, but the best test will be using the system.

Before the making the switch, we’ll clean up the following obsolete concept_reference_sources and associated mappings.

  • RxNORM Combo: delete mappings and source
  • SNOMED NP: Migrate to SNOMED CT with a mapping type of NARROWER THAN
  • ICD-10-WHO NP (and ICD-10-WHO NP2) : further analysis by myself and @akanter before removal
1 Like

Discussion today: Next phase of PIH implementation (before go-live) is to:

  • OCL Module Update: Need a new release to include the range of fixes add since November, i.e., most of Mike’s contributions.

  • Follow up on [OCLOMRS-1075] Improve creation of concept sources if necessary - OpenMRS Issues

  • Join @ball the week after next for some of her concept management workflow testing :heart:

  • have sources for the different implementations, e.g. PIH-Peru, PIH-Mexico. (That way a collection could have inputs from both the Golden PIH source and Implementation-specific sources; this mimics the current spreadsheet workflow; also allows site-specific organization/member control (e.g. give Peru team access to PIH-Peru source).) However, if this slows down initial go-live, then this can wait.


*An idea for the future (non-urgent, non-blocker):*
  • Review PIH concepts marked as “SAME-AS” to CIEL concepts, to check for ones that more belong as customized CIEL concepts. (i.e. they are exactly the same as the CIEL concepts, or just a slight non-breaking change)
    • But, really only adds value once we can say “FYI, this CIEL concept was updated; have a look”

My hope is we can get PIH to a place where the “truth” of PIH dictionaries are managed in OCL and flow out from there. I’m not sure it’s worth the effort to try keep a copy of PIH’s dictionary up to date (in real time) within OCL while it’s being managed outside of OCL. That said, during this testing phase, we can get you all in a position to perform your own imports (i.e., I could create a recipe for how to use the ocl_omrs script to generate an import json, delete what’s in OCL, recreate the source, import the json, and then recreate the collection).

1 Like

This would be very useful - thanks @burke !

1 Like

In today’s discussion of the mappings, we discussed removal of RxNORM Comb (not Combo). There are 37 concepts in the PIH dictionary with RxNorm Comb mappings. I don’t know why the OCL search doesn’t include all reference sources in the filter. @burke

image

For example: Cotrimoxizole

https://app.staging.openconceptlab.org/#/orgs/PIH/sources/PIH/concepts/916/

@akanter You suggested removing RxNORM Comb mappings, but if I did this for this example (Cotrimoxizole), there would be no RxNORM* mapping. Is that correct? If I look at the CIEL dictionary, that concept has an RxNORM reference term. I will check all 37 concepts with RxNORM Comb and make certain they have RxNORM mappings before I delete the RxNORM Comb mappings.

First, the RxNORM-Comb codes were CIEL-created codes for multi ingredient drugs. That was because RxNORM didn’t originally have maps for those. They now do. Many of these are already replaced (so in CIEL the PIH 916 concept has RxNORM 10831. Not all do, since some are international or are no longer available. For those, I tend to put multiple ingredient maps or use SNOMED. I think deleting the RxNORM-Comb is fine since they have no external meaning, but updating those concepts with the CIEL maps would make sense.

I’ve documented the recipe I use to import PIH into OCL staging and created a Google Doc to go along with this Talk thread where we can document remaining steps without the formality of an issue tracker (I’ll add this to the list of resources in the original post on this thread).

The mappings are there if you search specifically for “RxNORM-Comb” (ref) or if you look at mappings for specific concepts like Cotrimoxazole. I think the reason we don’t see RxNORM-Comb in the filter list is because only the top ten most used concept sources are listed. There’s a search feature in filters, but I think it just searches across the subset of filters listed (i.e., only searches within these top ten entries for sources, since it doesn’t reveal RxNORM-Comb).

This wasn’t obvious, but a clear explanation. Thanks @burke for creating the google doc.

Update: Today Ellen trialed using the OCL Term Browser to manage updates to concepts - she walked @jamlung and I through 2 examples of her workflow. We ran into a few blocking bugs.

2 main blockers: (@jamlung will you ticket these in oclissues or would you like me to?)

  • Can’t save updates in TermBrowser: Could not save updates to a concept due to error message saying “Invalid Name Type… An Error Occurred While Saving New Concept Version”
  • No UI to specify Set Members, Answers, and Mappings in TermBrowser

Recording: Testing PIH Concept Management Use Cases for OCL Termbrowser & Dictionary Manager (Feb 9 2022) - Indiana University

Notes/findings: OCL UAT with Ellen for PIH 2022-02-09 - Google Docs

We need a solid mechanism for creating the OCL ID in the OpenMRS/OCL Dictionary Manager (and eventually the OCL Term Browser) for every new concept in the PIH dictionary. Along with @burke, we decided to use the PIH numeric mapping as the OCL ID when we import the PIH dictionary into OCL. This number is unique and currently matches the concept_id on the PIH concept server. But when we move to OCL, there should be an easy way to automatically create/increment the OCL ID with the next PIH value. It would also be helpful to automatically save this as a reference term and concept mapping.

This is essential since we use this PIH mapping for forms and reports.

Does this deserve a JIRA ticket? How should this work for non-PIH dictionaries?

Thanks @grace @jamlung for today’s helpful exercises. Three minds working together = very productive.

1 Like

Thanks @grace, I have them recorded and will go over them with Jon tomorrow. We will want to group the issues from yesterday and then I will create the relevant tickets from that.

1 Like

@ball I am going to talk with Jon about this tomorrow. I am not sure if this is something that OCL should handle or not. We’ll update you on the next OpenMRS Squad call.

1 Like