There were very few duplicates in the system (less than 10). I dont know where the validation is being done … is this the new OWA app ?
Here is a basic summary of what i found
Some cases the duplicate name was the preferred name in one concept and a synonym in another concept which is/should be allowed
Where the duplicate name was the default / preferred name in two different concepts (and this should fail validation), you will find that one of the concepts was retired (and this should pass)
In some cases i found no duplicate so i am not sure how this came about
See detailed examples and explanations on the above in the attached file
The French name “Varicelle” for concept 892 is correct. The French name for CIEL:123358 should be changed from “Varicelle” to something else, perhaps “infection par herpès zoster”.
Likewise, @akanter’s recommended corrections for CIEL:162305 to “klas eparinn” is good. And CIEL:77413 can remain with the name “eparinn”.
In Haitian Creole, itching (CIEL:136455) and rash (CIEL:512) are generally both referred to as “Gratèl”. Maybe rather than voiding the itching ht name, we could modify rash (CIEL:512) concept. Set the fully specified name to “Maladi po” which literally translates “skin sickness” and set the synonym to “Gratèl”.
@judy, from your attachment (and the fact that you’re referring to tags
instead of concept_name_type) I see you are doing the concept management in
a pre-1.7 version of the OpenMRS API. So what we’re seeing here is a
combination of the upgrade process and validation.
Any chance that you can revisit one of your examples of 2 and 3 (e.g. the
ones you don’t think should fail validation) and see how they look in a
1.7+ version of CIEL?
We maintain the dictionary in 1.6.6 and then have scripts that generate releases for all other versions … How we are able to provide concepts for everyone … Ideally I guess we should figure out a way to make it agnostic and independent of openmrs versions maybe using ?rest as not all users deploy openmrs @akanter can comment more on this
Darius,There may be an issue in the upgrade process then, as we are not seeing those duplicates (mostly) on the pre1.7 side. We probably can check one of the later versions, but will someone else check out to see where the problem is in the upgrade?
Do you agree with our interpretations of the validation checks? Why did this just come up? We have been releasing this content for some time.
Andy
Andrew S. Kanter, MD MPH FACMI
Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Andy, I agree it must be an issue in the upgrade process; I don’t know why this is just turning up now. (Mostly likely something to do with this being the first time that we’re releasing a reference application version on top of platform 1.11.x.)
If Judy can look closely at one of each type of failure, and highlight what differs from the 1.6 version (that she says is okay) and the 1.11 version (that is failing), then we can get a dev to take it from there.
I think one major issue is that we have some concepts with ONLY synonyms in Spanish. One of them needs to be elevated to the preferred/default position and that cannot be the same as an existing preferred/default concept name. I am thinking we may need to remove synonyms where we don’t have an adequate translation of the default description/concept_name.
What do you think should happen for this? should a new CIEL version be released? or should the values be ignored/manually corrected? to make this release happen?
We are nearing a month past our release date, If the “Correct Fix” takes more than a week, it would be preferable to have a workaround in our release. The “Correct Fix” can either be released in a maintenance release or the next release
One thing that will speed things up: it is ok for UAT to begin with James’s
hackily-cleaned-up version of the dictionary. We should have a final
release of CIEL for our actual refapp 2.4 release though.
Even with the clean up hacks of the CIEL, after metadatasharing successfully validates and exports the packages, referenceapplicationmetadata chokes on the exported concepts header files, see error.
This is why I say there may be problems with validation of the concept dictionary. If the concept dictionary passes validation in metadatasharing it should also pass validation in referenceapplicationmetadata.
In any case this is also a blocker on the release of refapp 2.4.
I have just realized I am making a mistake with process.
The packages were exported with 1.11 CIEL cleaned up but I was running referencemetadata with a different database/CIEL. I have to dump the cleaned up CIEL from mdsbuilder and import it to my local openmrs before I run the build to get rid of the last error I mentioned.
I have looked at the code and confirmed that mds and referencemetadata use the same validation code.
I am working on this and will provide an update soon. I will be able to probably confirm that the initial set of errors are the only errors which need to be addressed.
We believe we have identified at least one issue. There are concepts with 2 Spanish synonyms but no Spanish FSN/default names in 1.6.6. One of these is propagated to the default in the UI in 1.6.6 but not in the database. I think these are being actually migrated as preferred in the migration script and causing errors. I am working with Judy on fixing them, but we might just remove them for now. I want to make sure that I fully understand the query being used for validation and which errors absolutely have to be fixed. We fixed the ones in pastebin, I believe, but @judy has to confirm. I will try to put out a release either tonight or tomorrow (but it sounds like you might have a version you can use now)
I was not following the process for release of CIEL correctly. Once the errors mentioned in the pastern are fixed we are good. My manual removal of the errors worked fine for now.
As for the validation rules, a /dev/5 will have to provide those.
I am posting an 1.11.x version of the release in the dropbox for testing purposes. Please run it and see if it is OK. If it is, I will make it the release, if not, then we will fix whatever you tell us in PasteBin.
By investigation is as Andy pointed out … Try the new dictionary and share the error logs again and I will work to fix them in time for the release … The reason why we paused was that we felt that the validation rules (which we are still waiting for ) are inconsistent … Look forward to the testing
Is there still an issue where some concepts are being upgraded incorrectly from 1.6 to 1.11? As I understand it, you said some of the errors mentioned by James in the pastebin are of this type.
If that is the case, we are waiting on you to give a specific example, so we can look into the problem. (But we cannot easily review the entire concept upgrade process in general.)
I think no team has enough bandwidth to check the whole validation … Let james use the new release that has the errors we identified fixed and if there are issues we can look into them …