Moving RefApp concept management into OCL

(This was an email, but I’m copying and pasting it here for broader visibility.)

I have attempted to define our list of concepts that are included in the reference application in an OCL collection here: https://openconceptlab.org/orgs/OpenMRS/collections/RA/concepts/

@jteich , @burke , can you poke around at this just so you know what there, and get a sense of the OCL UI? (FYI the current list of diagnoses in the reference application is horribly inadequate. It’s the exploded set-of-sets of ICPC Diagnosis Categories. To follow this in the OCL UI you have to click on one of the set members and then click on the Mappings tab.)

@paynejd , the fact that I felt compelled to write the last sentence above makes me agree that we must prioritize the UI ticket about browsing between concepts, and/or a simpler one about having a better default “details” screen.

Currently these concepts are packaged in the referencemetadata module in three xml files:

  • Reference_Application_Concepts-20.xml
  • Reference_Application_Order_Entry_and_Allergies_Concepts-15.xml
  • Reference_Application_Diagnoses-8.xml

(I don’t know where these xml files come from; they used to be MDS packages in http://mdsbuilder.openmrs.org:8080/openmrs but I guess Rafal has changed that around.)

I wrote a script that will take as input a list of CIEL concept IDs and update the OCL collection. (This is currently necessary since the UI is too awkward to use at this scale, and there are actually 891 individual references in the collection, meaning concepts and mappings.)

I will check this script into github later, so that anyone else may use it later as part of the release. [Edit: I checked this into github at https://github.com/djazayeri/openmrs-contrib-updateoclcollection]

@raff, can you plan (eventually) to get the team working on incorporating this into the refapp installation/startup process, and remove the xml files?

CC @akanter @ssmusoke @ball

The for putting together this summary, @darius. You’re correct that the current list of diagnoses in the reference application is lacking. My hope was that we could come up with something similar to SNOMED CT-CORE, which contains about 6000 of the most useful diagnoses and findings. Glancing at the RA Concepts collection:

  • It’s a bit confusing to me. The RA Concepts collections contains 18 diagnoses, but when I look at ICPC Diagnosis Categories, it’s contains about 17 concept sets of various diagnoses, which go well beyond the 18 diagnoses shown in the RA collection. Is there anywhere to see & search the diagnoses you’re proposing in a single list? Or only by clicking through each of the ~17 concept sets under ICPC Diagnosis Categories?

  • Inclusion of just 2 symptoms or findings (Bronchospasm and Rash) is inadequate, IMHO. In clinical care, we don’t just make diagnoses. Many “diagnoses” being treated are undiagnosed symptoms & findings, like Abdominal pain, Chest pain, Anxiety, Fatigue, Cough, Fever, Weight loss, etc. SNOMED CT-CORE includes nearly 1000 findings. Maybe these are buried under the ICPC Diagnosis Categories?

I was testing importing concepts and mappings and found out that some mapping are referring to concepts that are not being imported. As @raff suggested I’m pasting this list of mappings here http://pastebin.com/7S5Uhpuv

@darius, the issue with mappings listed by Adam is that e.g. https://openconceptlab.org/orgs/CIEL/sources/CIEL/mappings/583f3abd65baaa000771c5f0/ refers to https://openconceptlab.org/orgs/CIEL/sources/CIEL/concepts/160168/ which doesn’t seem to be included in the RA collection… Any idea why it’s not included?

Sorry, there’s a bug in my script. Let me fix it, and then we can try this again.

I fixed the bug in my code. Now there are 2316 references in the collection, instead of the 891 I was getting before.

The production OCL server is down (hopefully not because I broke it!), so I have set this up on the staging server: https://staging.openconceptlab.org/orgs/OpenMRS/collections/RA/concepts/

Here is what things look like now:

(I did not go and check all the mappings mentioned by @adamg but I did verify that concept 160168 mentioned by @raff is included now.)

Please take another look. :slight_smile:

The OCL production server is back, so you can now see this at https://openconceptlab.org/orgs/OpenMRS/collections/RA/concepts/

Thanks Darius.

Unfortunately when I try to create a new version of the collection from the UI the server times out with “504 Gateway Time-out”. It probably needs to be fixed to be run as a background task…

@paynejd, do you know of any workaround we could use here? Maybe there’s a way to increase the time out…

@raff when I look at https://openconceptlab.org/orgs/OpenMRS/collections/RA/versions/ it seems that creating the versions actually did work, it just took forever and did happen in the background (but the page timed out first so you had no way of knowing this).

I had created a related ticket at https://github.com/OpenConceptLab/oclapi/issues/481 so maybe you should add a comment to it.

Great! @adamg, how about you try again now?

@raff There are still some mapping errors like before but for different ones this time. See below

@adamg, it doesn’t help much since we can’t see the reason of failure. Can you work on displaying the reason of failure by the OCL module? Meanwhile, what you see in server logs?

There are no errors in the server logs. I’ve created issue to fix logging. OCLM-60

@raff @darius Updated screen shot with error messages

When I do both of these searches I do see the concept included in the RA collection:

Help me understand what the problem is.

I’m trying to find the problem, but you are right that these concepts are in the collection. For some reason they are not imported…

@darius, interestingly those concepts are not included in the export json file for the version I created at https://api.openconceptlab.org/orgs/OpenMRS/collections/RA/v2.6.0-alpha.2/export/

I wonder if it is an issue with the export logic or the fact that 505 is thrown at some point…

(I’m creating a new version v2.6.0-alpha.3 to check if it wasn’t due to some indexes not being up to date. Update: v2.6.0-alpha.3 has the same isssue)

@paynejd, do you have any idea why it might be that some concepts are missing (see Darius last post) in the export file, but are visible in the UI? Were those concepts added properly to the collection or is this a bug in export code?

Hi Rafal, there have been a few experiences in the past where a resource is added to the repository but not the repository version, or vice versa. However, we have not been able to isolate when or where the bug is occurring. Does this seem like what might be happening in your situation? If this is it, it is possible to fix it at the MongoDB level and then update the index.

Thanks! Yes, I believe it’s the case. Can you fix that in MongoDB?