CIEL: Duplicate names in 1.11.x for Eparinn & Gratel in 'ht' locale

@darius and @jdegraft cc @akanter

I spent some time figuring out why you got all the errors you listed here : http://pastebin.com/Wy2BAqqD

There were very few duplicates in the system (less than 10). I dont know where the validation is being done … is this the new OWA app ?

Here is a basic summary of what i found

  1. Some cases the duplicate name was the preferred name in one concept and a synonym in another concept which is/should be allowed

  2. Where the duplicate name was the default / preferred name in two different concepts (and this should fail validation), you will find that one of the concepts was retired (and this should pass)

  3. In some cases i found no duplicate so i am not sure how this came about

See detailed examples and explanations on the above in the attached file

Happy to hear any other suggestions

Investigation.pdf (416.3 KB)

1 Like

Glad you’re finding and fixing these problems.

The French name “Varicelle” for concept 892 is correct. The French name for CIEL:123358 should be changed from “Varicelle” to something else, perhaps “infection par herpès zoster”.

Likewise, @akanter’s recommended corrections for CIEL:162305 to “klas eparinn” is good. And CIEL:77413 can remain with the name “eparinn”.

In Haitian Creole, itching (CIEL:136455) and rash (CIEL:512) are generally both referred to as “Gratèl”. Maybe rather than voiding the itching ht name, we could modify rash (CIEL:512) concept. Set the fully specified name to “Maladi po” which literally translates “skin sickness” and set the synonym to “Gratèl”.

Keep up the good work.

@judy, from your attachment (and the fact that you’re referring to tags instead of concept_name_type) I see you are doing the concept management in a pre-1.7 version of the OpenMRS API. So what we’re seeing here is a combination of the upgrade process and validation.

Any chance that you can revisit one of your examples of 2 and 3 (e.g. the ones you don’t think should fail validation) and see how they look in a 1.7+ version of CIEL?

We maintain the dictionary in 1.6.6 and then have scripts that generate releases for all other versions … How we are able to provide concepts for everyone … Ideally I guess we should figure out a way to make it agnostic and independent of openmrs versions maybe using ?rest as not all users deploy openmrs @akanter can comment more on this

I will try this a little later

Darius,There may be an issue in the upgrade process then, as we are not seeing those duplicates (mostly) on the pre1.7 side. We probably can check one of the later versions, but will someone else check out to see where the problem is in the upgrade? Do you agree with our interpretations of the validation checks? Why did this just come up? We have been releasing this content for some time.

Andy Andrew S. Kanter, MD MPH FACMI

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology Columbia University

Andy, I agree it must be an issue in the upgrade process; I don’t know why this is just turning up now. (Mostly likely something to do with this being the first time that we’re releasing a reference application version on top of platform 1.11.x.)

If Judy can look closely at one of each type of failure, and highlight what differs from the 1.6 version (that she says is okay) and the 1.11 version (that is failing), then we can get a dev to take it from there.

I think one major issue is that we have some concepts with ONLY synonyms in Spanish. One of them needs to be elevated to the preferred/default position and that cannot be the same as an existing preferred/default concept name. I am thinking we may need to remove synonyms where we don’t have an adequate translation of the default description/concept_name.

Hello @judy, @akanter, @darius ,

This is a blocker for Releasing OpenMRS 2.4

What do you think should happen for this? should a new CIEL version be released? or should the values be ignored/manually corrected? to make this release happen?

We are nearing a month past our release date, If the “Correct Fix” takes more than a week, it would be preferable to have a workaround in our release. The “Correct Fix” can either be released in a maintenance release or the next release

also inviting dev/5/'s to request their opinion on how to proceed: @burke, @wyclif, @dkayiwa, @mseaton

One thing that will speed things up: it is ok for UAT to begin with James’s hackily-cleaned-up version of the dictionary. We should have a final release of CIEL for our actual refapp 2.4 release though.

-Darius (by phone)

Even with the clean up hacks of the CIEL, after metadatasharing successfully validates and exports the packages, referenceapplicationmetadata chokes on the exported concepts header files, see error.

This is why I say there may be problems with validation of the concept dictionary. If the concept dictionary passes validation in metadatasharing it should also pass validation in referenceapplicationmetadata.

In any case this is also a blocker on the release of refapp 2.4.

Can @raff, @wyclif, @dkayiwa, out someone else please put resolving this blocker at the top of their priority list?

@raff has done the most historically with concept validation. @wyclif is in the closest time zone to @jdegraft.

Who can step up and help with this?

-Darius (by phone)

1 Like

I also need clarity on what errors are really errors so they can be fixed.

Andy

Andrew S. Kanter, MD MPH FACMI

I have just realized I am making a mistake with process.

The packages were exported with 1.11 CIEL cleaned up but I was running referencemetadata with a different database/CIEL. I have to dump the cleaned up CIEL from mdsbuilder and import it to my local openmrs before I run the build to get rid of the last error I mentioned.

I have looked at the code and confirmed that mds and referencemetadata use the same validation code.

I am working on this and will provide an update soon. I will be able to probably confirm that the initial set of errors are the only errors which need to be addressed.

@akanter, I believe we are still waiting for @judy to follow up on this:

Per CIEL: Duplicate names in 1.11.x for Eparinn & Gratel in ‘ht’ locale I guess she’ll get to it when she has time…

We believe we have identified at least one issue. There are concepts with 2 Spanish synonyms but no Spanish FSN/default names in 1.6.6. One of these is propagated to the default in the UI in 1.6.6 but not in the database. I think these are being actually migrated as preferred in the migration script and causing errors. I am working with Judy on fixing them, but we might just remove them for now. I want to make sure that I fully understand the query being used for validation and which errors absolutely have to be fixed. We fixed the ones in pastebin, I believe, but @judy has to confirm. I will try to put out a release either tonight or tomorrow (but it sounds like you might have a version you can use now)

I was not following the process for release of CIEL correctly. Once the errors mentioned in the pastern are fixed we are good. My manual removal of the errors worked fine for now.

As for the validation rules, a /dev/5 will have to provide those.

1 Like

I am posting an 1.11.x version of the release in the dropbox for testing purposes. Please run it and see if it is OK. If it is, I will make it the release, if not, then we will fix whatever you tell us in PasteBin.

By investigation is as Andy pointed out … Try the new dictionary and share the error logs again and I will work to fix them in time for the release … The reason why we paused was that we felt that the validation rules (which we are still waiting for ) are inconsistent … Look forward to the testing

1 Like

Hi @judy,

Is there still an issue where some concepts are being upgraded incorrectly from 1.6 to 1.11? As I understand it, you said some of the errors mentioned by James in the pastebin are of this type.

If that is the case, we are waiting on you to give a specific example, so we can look into the problem. (But we cannot easily review the entire concept upgrade process in general.)

I think no team has enough bandwidth to check the whole validation … Let james use the new release that has the errors we identified fixed and if there are issues we can look into them …