OCL (and connector) annoyances and problems

darius · December 16, 2017, 6:08pm

I just tried creating a concept in OCL to sync to OpenMRS, and I ran into some issues that I consider blockers for asking normal users to start using this.

The OCL UI gives you the option to say datatype=“None” but if you choose it the concept will fail to import into OpenMRS. (You have to manually type “N/A” which is not an option in the dropdown.) I believe that our connector module should transparently convert datatype of “None” to “N/A” if that’s what OCL prefers.
The default concept name type in the OCL UI is “Fully Specified” but unless you manually change this to “FULLY_SPECIFIED” the concept will fail to import into OpenMRS. I believe that our connector module should transparently convert this.
I need a way to, given a collection, update it to include the newest versions of concepts. (Second best would be if at least I can add a newer version of a concept to a collection and have it automatically replace the older version.)
In the OCL subscription module, after a failed import, I see the message “Your last import ended with errors. Do you want to run that import again or ignore all errors?”, but I don’t exactly understand what this means. Does the second option mean something like “skip the last import and download updates again”?
The subscription module appears to discard any error messages whenever I try to import again and get an error.

E.g. Here is the first error:

But when I get another error the previous error instead shows “0 items updated”, and error_message has actually been nulled out in the DB:

subscribedToSnapshot doesn’t seem to work the way I expect. I would expect that if I subscribe toa collection with subscribedToSnapshot=true then I don’t have to create a new version of the collection to be able to pull new updates. But it doesn’t seem to work like that.
After setting subscribedToSnapshot to true, and then back to false I’m getting errors. (I don’t know if that’s the cause.) This error means that fetchLatestOclReleaseVersion is returning null for https://openconceptlab.org/users/djazayeri/collections/TEST (Now that I type this I wonder if the current failure is actually expected, since I’ve never made a released version; but if that’s true I don’t know how it worked before.)

Caused by: java.lang.NullPointerException at org.openmrs.module.openconceptlab.client.OclClient.fetchLastReleaseVersion(OclClient.java:130)

As mentioned on Integrating openmrs-module-openconceptlab with Bahmni - #34 by darius the latest builds of the OCL subscription module are broken for me.

darius · December 16, 2017, 6:19pm

I forgot two:

There’s now way to “Add a set member”; I have to do an annoying amount of clicking to add a mapping whose type is concept-set (and the autosuggest widget for map type doesn’t seem to work).
Something is wrong with the locale dropdowns. They show lots and lots of duplicates.

darius · December 17, 2017, 7:06pm

I stepped through the code and I can state that however exactly (4) is intended to work, it’s not doing the right thing.

Even if I choose the “run again” options, it does not run it again.

This is because the REST call to start a new import clears the error message from the prior import.

the line getImportService().ignoreAllErrors(anImport) in ImportResource.save(Import) is always called
then later in Importer.runTask the line Import lastImport = importService.getLastSuccessfulSubscriptionImport() always returns something even if the last import was unsuccessful in real life because starting a new import would have cleared the error message.

Thus when I re-run an import which failed and should fail again, it instead “works” though it doesn’t import anything.

darius · December 20, 2017, 2:16am

Two more annoyances:

First, when I add a concept to a collection, I have to option to automatically add its mappings, but it doesn’t automatically add concepts that those mappings refer to. That’s annoying. (Or maybe it’s because I’m trying the use case where I create a “Dispositions” set in my own dictionary whose set members are CIEL concepts.)

Second, the subscription module will fail to import a concept that doesn’t have an externalId defined in OCL. I understand that this is populated automatically when an OpenMRS dictionary like CIEL is imported into OCL, but if I’m creating a new concept in OCL it’s a pain to generate a UUID in some external tool and paste it in.

One approach would be that when importing a concept with no externalId defined, then we use OCL’s uuid as the uuid for our concept. (We’d have to handle the case where after doing this someone adds an externalId in CIEL, which might be a pain.)

But stepping back, I see a whole lot of UI annoyances, and taken together I don’t see how I can possible recommend that any normal user actually use OCL to manage their OpenMRS dictionary. So @paynejd, I wonder if there is a way that we can include a UI in OCL for simplified concept management specifically for the OpenMRS use case (by constraining what’s allowed, and letting the UI refer to some specific OpenMRS constructs). Can we do this in some way that doesn’t destroy your vision of OCL as agnostic of what tool it’s used with? (e.g. as a user I turn on “OpenMRS mode” for a source or collection). Otherwise I think we’ll have to write some external app to manage dictionaries in OCL via its REST API.

@raff, any reason I shouldn’t add these substitutions? I would put them in org.openmrs.module.openconceptlab.importer.Saver#toConcept and org.openmrs.module.openconceptlab.importer.Saver#updateOrAddNamesFromOcl, cool?

raff · December 20, 2017, 10:25am

Yes, go ahead.

raff · December 20, 2017, 11:06am

We have an issue for that here: https://github.com/OpenConceptLab/DATIM-Metadata-Project/issues/142 It’s included in the DATIM project scope so it’s scheduled to be worked on.

It shouldn’t be displayed when subscribing to versions. It was intended for subscribing to SNAPSHOTs, where the module downloads a diff of changes since the last import. If you have errors you will most likely want to import that diff again. However, if you are not able to fix those errors or you don’t care about them, you will most likely want to stop trying importing that diff again and continue with a new diff (the second option).

There’s a way to set an OpenMRS validation scheme for a given source or collection. It feels natural to extend that concept to not only validation, but also defining dropdown values.

I will be looking into issues on the module side and release fixes by the end of this week.

paynejd · December 20, 2017, 7:52pm

Hey @darius Great feedback and thanks for testing out the subscription module I think that everyone who has interacted with OCL agrees that UI work is needed! Users to date have all imported content and used the UI only for small updates (which has been tolerable at best), but that definitely doesn’t work for the vast majority of users out there. I have a general comment on a possible approach followed by specific responses to your points.

The general comment is that it appears that a UI specific to a use cases may not be a unique requirement to OpenMRS, as this also came up DHIS2, since managing indicators and their disaggregates have some unique workflows. I think as a community we should be open to that, but probably only if necessary. OpenMRS represents the bread-and-butter use case in terms of concept management, and it sure seems like it should be supported out of the box and that non-OpenMRS dictionary managers would be benefit from that functionality as well. We’ll just need to strike the right balance between generic (which may take longer) and use-case-specific.

We already have a the OpenMRS Custom Validation Schema, that validates against specific lookup options and ensures name uniqueness across a repository for a given locale. A few additional generic fixes may get us where we need to get, and we could consider for each whether the generic or use case specific approach makes the most sense:

Repository-level control of lookup options – e.g. each repo could select a repository in OCL that contains the lookup options; includes the ability to select which “name type” is included in the dropdown
Ability to automatically set lookup options by choosing a custom validation scheme
Additional control over how references to mappings and toConcepts are added to a collection
Semi-automated updating of references in resources to latest version (includes changes in dependencies)

Are there other changes that need to be added to the above list?

Specific responses:

OCL stores the OpenMRS specific synonyms, i.e. N/A (see here: https://www.openconceptlab.org/orgs/OCL/sources/Datatypes/concepts/None/), but the UI and subscription module do not have any way to take advantage of that yet. The solution you proposed seems right for the time being.

Same as #1 above, except that OCL does not store this as a synonym. We could easily add that if it made sense.

As @raff said, we have a ticket for automating the “update collection references to latest” functionality. Your alternative approach may be a good one to consider in addition.

Agree with this – using the UI to create mappings is challenging and error-prone. There should be a UI method adding mappings directly in the concept editor, which would also address the concept set issue.

Could you point out some examples and indicate which form? I looked for examples of this but did not find any.

You described the OCL behavior correctly – if you add a concept, you can select the checkbox to automatically add any mappings stored in the same source for which the selected concept is the “fromConcept”. This behavior is controlled using a “cascade” URL parameter, which can be set to either none (default for the API) or sourcemappings (default for the Web App). For example:

PUT /orgs/KenyaMOH/collections/KenyaEMR/references/?cascade=sourcemappings

We should be able to add additional options for the cascade parameter that determine whether “toConcepts” are also added, and whether those toConcepts must be in the same source or if they can be external. As an example, if you think about the situation in which OCL also hosts ICD-10, you may only want to add CIEL concepts but keep ICD-10 concepts external.

darius · December 21, 2017, 12:41am

I can take care of the trivial fixes (like text substitutions in the connector module) if you can look into some of the things that require more knowledge of OCL and the existing module design.

One suggestion is that we can just delete these 4 lines (since there’s a separate ImportActionResource for ignoreAllErrors): https://github.com/openmrs/openmrs-module-openconceptlab/blob/1.2.2/omod/src/main/java/org/openmrs/module/openconceptlab/web/rest/resources/ImportResource.java#L58-L61

darius · December 21, 2017, 3:46am

Where I’m coming from is that:

I want to provide a UI to support the common use case we’ve been discussing for years of managing your OpenMRS dictionary in OCL
I want the UI for common actions to be simple
I want to be able to build something quickly and iterate on it

So, yes, some things could be approached generically, but if that will take longer and/or create a worse UX for this use case, I don’t think this is the right way to to go at this point.

For example I would like a simplified UI to “create a value set (in my own source) that contains CIEL concepts”. (E.g. I would have my own “TB program outcomes” set that’s custom to me, but whose members are normal CIEL concepts.)

Another example, instead of “a UI method adding mappings directly in the concept editor” I want the UI to just let you “add a member (to a set)” or “add an answer (to a question)”.

And, I want to hide the externalId (or else call it “OpenMRS UUID” and autogenerate it).

All of these improve the UX for managing an OpenMRS dictionary in OCL, but I realize they’re not generic. If it’s not appropriate to put this kind of thing in the OCL UI I’m open to building this in a separate app. I’ll take your direction on which approach is more appropriate.

I went to https://openconceptlab.org/users/djazayeri/collections/new/ and clicked on the dropdown for Default Locale, then scrolled down a bit and I see this:

Also, typing “en” or “English” doesn’t autocomplete or anything.

raff · January 3, 2018, 12:43pm

I addressed issues with converting None to N/A and Fully Specified to FULLY_SPECIFIED when importing into OpenMRS.

I have also addressed the issue with errors being always ignored.

The update to snapshots feature is still broken for collections as the end point on the OCL server is not returning any results. I will be looking into a fix before releasing a new version of the module (in case it requires changes on the module side as well).

paynejd · January 4, 2018, 7:15pm

@raff What is the specific issue you are experiencing with the collections endpoint not working? We are making heavy use of the collections API endpoints in the DATIM metadata sync script without any errors.

@darius Do you have ideas where this functionality would go in the OCL UI? I am personally okay with this being part of the OCL UI, and I’d like to hear from others too. Let’s draft up how this would work and get some community feedback.

Also, I created a new ticket for the duplicate locale issue: https://github.com/OpenConceptLab/DATIM-Metadata-Project/issues/194

raff · January 5, 2018, 11:24am

@paynejd, see e.g. https://api.staging.openconceptlab.org/users/rafal/collections/RA/?includeMappings=true&includeConcepts=true&includeRetired=true which is not including concepts nor mappings fields…

paynejd · January 5, 2018, 1:44pm

@raff I was able to reproduce the error with the link you sent, and it also did not work for sources: https://api.staging.openconceptlab.org/orgs/PEPFAR/sources/MER/?includeMappings=true&includeConcepts=true&includeRetired=true

Does export generation rely on this feature working? If yes, that means exports would only be working for cached exports of repo versions and would not work for any new content.

darius · January 10, 2018, 9:05pm

One idea is to let a user indicate, for each source:

which “UI Flow” to use (in my case “OpenMRS”)
which “trusted dictionary” to prefer (in my case CIEL)

Then, when viewing your source, you’d have some additional shortcuts where the New Concept button is now, to do things like:

create a Diagnosis (defaults datatype to none, class to Diagnosis, lets you give ICD10 and SNOMED mappings on the first screen; searches CIEL for similar concepts in the background as you’re creating it)
create a Set of concepts from my trusted dictionary (datatype=none, class=ConvSet, lets you search for concepts in your trusted dictionary and add only these to the set)

I’d like a shortcut to create a special kind of collection that prefers to take concepts from 1-n trusted dictionaries (in my case CIEL followed by Bahmni Default Concepts), and also includes all concepts from 1-n sources (e.g. my implementation-specific concept source).

This would have a special UI flow for including concepts from the trusted dictionaries (and automatically pulling in their mappings), and it would automatically include all concepts and mappings from its include-all sources (maybe this happens with a scheduled task, maybe it happens when you trigger a new version).

Basically, what I’m getting at is that I want to make it easy for someone who doesn’t care to really learn about all the complexity and power of OCL to be able to do a small number of specific workflows that have to do with creating a few custom concepts, and pullings lots of concepts from CIEL, and some Bahmni curated lists, and merging these all together into a collection. Honestly the best UI would probably come from doing a purely custom app for this, but if it helps to put this in the main UI, we can do it there too…