[SOLVED] Generation of Meta-data like Reference Application for Concept Installation

ssmusoke · January 5, 2016, 10:15am

I am trying to leverage meta-data sharing of data like the reference application in the referencemetadata module to enable automatic configuration of encounters, concepts, locations. My question is how is this data generated

Concepts - https://github.com/openmrs/openmrs-module-referencemetadata/blob/master/api/src/main/resources/Reference_Application_Concepts-17.xml

Thanks in advance

pascal · January 5, 2016, 10:27am

I’m not sure, but I searched the repo and based on this it seems to be using dataexchange’s DataImporter.

ssmusoke · January 5, 2016, 11:11am

Thanks @pascal this seems to work perfectly, as I have duplicated concepts so the metadata sharing module was failing to export them and I cannot change the concepts at this time.

@pascal Can I assume that the zip files are generated using metadata sharing module?

An interesting question to @raff What is the future of the data exchange module, is it going to be integrated into the metadata sharing? Will it be extended to include encounters, and other metadata?

pascal · January 5, 2016, 11:17am

I’m not sure.

raff · January 5, 2016, 12:05pm

Stephen, the Data Exchange module was just a hack to preserve concept IDs from CIEL when importing them into the Reference Application. The Metadata Sharing module does not let you do that. The Data Exchange module is going to be deprecated as soon as openconceptlab.org is ready for production. See https://wiki.openmrs.org/x/XYA5B

At this point I would recommend you to use https://wiki.openmrs.org/display/docs/Metadata+Deploy+Module

ssmusoke · January 5, 2016, 3:48pm

@raff thanks for the info about deprecation. I am looking for a way to export my concepts into an XML file that can be version controlled, metadata sharing cannot do that because of duplicated concepts hence cannot validate

raff · January 5, 2016, 4:17pm

If you really want to have duplicate concepts in your dictionary then maybe, you could try backporting https://issues.openmrs.org/browse/TRUNK-4626 and disabling validation for the time of export/import. It’s just an idea, not a recommendation

ssmusoke · January 5, 2016, 4:45pm

@raff will look into that, but yes I am stuck with duplicate concepts (legacy issues) for now, not sure if metadata sharing allows for export without validation @mseaton @mogoodrich any ideas here?

ulrich · January 5, 2016, 5:11pm

This reminds me of the errors I had. After doing a successful upgrade, when I installed the ref app modules it will not stop talking about duplicate concepts.

ssmusoke · January 5, 2016, 5:40pm

@raff I am running into an error with the data importer

http://pastebin.com/bECkvTbj

I tried it on a blank OpenMRS database (new install with tables) and also from the database in which I ran the export and the error is the same. Does the importer update existing concepts it the UUIDs or ids match existing ones.

darius · January 5, 2016, 6:37pm

Generally, the recommended approach for forcing multiple servers to have the same configuration, and having this configuration on source control, is to use the Metadata Deploy module.

You can see an example of how PIH uses it here:

github.com

PIH/openmrs-module-pihcore/blob/master/api/src/main/java/org/openmrs/module/pihcore/deploy/bundle/haiti/mirebalais/MirebalaisLocationsBundle.java

package org.openmrs.module.pihcore.deploy.bundle.haiti.mirebalais;

import org.openmrs.Location;
import org.openmrs.LocationAttribute;
import org.openmrs.module.metadatadeploy.bundle.AbstractMetadataBundle;
import org.openmrs.module.metadatadeploy.bundle.Requires;
import org.openmrs.module.pihcore.deploy.bundle.core.LocationAttributeTypeBundle;
import org.openmrs.module.pihcore.deploy.bundle.core.LocationTagBundle;
import org.openmrs.module.pihcore.metadata.haiti.mirebalais.MirebalaisLocations;
import org.springframework.stereotype.Component;

;

@Component
@Requires({ LocationTagBundle.class, LocationAttributeTypeBundle.class} )
public class MirebalaisLocationsBundle extends AbstractMetadataBundle {

    @Override
    public void install() throws Exception {

This file has been truncated. show original

github.com

PIH/openmrs-module-pihcore/blob/master/api/src/main/java/org/openmrs/module/pihcore/metadata/haiti/mirebalais/MirebalaisLocations.java

/**
 * The contents of this file are subject to the OpenMRS Public License
 * Version 1.0 (the "License"); you may not use this file except in
 * compliance with the License. You may obtain a copy of the License at
 * http://license.openmrs.org
 *
 * Software distributed under the License is distributed on an "AS IS"
 * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
 * License for the specific language governing rights and limitations
 * under the License.
 *
 * Copyright (C) OpenMRS, LLC.  All Rights Reserved.
 */

package org.openmrs.module.pihcore.metadata.haiti.mirebalais;

import org.openmrs.module.metadatadeploy.descriptor.LocationAttributeDescriptor;
import org.openmrs.module.metadatadeploy.descriptor.LocationDescriptor;

import java.util.Arrays;

This file has been truncated. show original

You’ll notice that by default the metadata is described in Java code, rather than a CSV, like you might prefer. In KenyaEMR they use CSV with metadatadeploy for their Kenya master facility list (which was long).

The underlying code for that is:

github.com

openmrs/openmrs-module-metadatadeploy/blob/10fe2bb072dbbe6a8b91a97d20f3b00c8ee7b022/api/src/main/java/org/openmrs/module/metadatadeploy/bundle/AbstractMetadataBundle.java#L128


			deployService.uninstallObject(outgoing, reason);
		}
	}


	/**
	 * Performs the given synchronization operation
	 * @param source the object source
	 * @param sync the synchronization operation
	 * @return the synchronization result
	 */
	protected <T extends OpenmrsMetadata> SyncResult<T> sync(ObjectSource<T> source, ObjectSynchronization<T> sync) {
		MetadataSynchronizationRunner<T> runner = new MetadataSynchronizationRunner<T>(source, sync);
		return runner.run();
	}


	/**
	 * Fetches a possibly existing object (non fail-fast)
	 * @param clazz the object's class
	 * @param identifier the object's identifier
	 * @return the object or null
	 */

KenyaEMR’s code to use this underlying function is:

github.com

I-TECH/openmrs-module-kenyaemr/blob/master/api/src/main/java/org/openmrs/module/kenyaemr/metadata/sync/LocationMflCsvSource.java

/**
 * The contents of this file are subject to the OpenMRS Public License
 * Version 1.0 (the "License"); you may not use this file except in
 * compliance with the License. You may obtain a copy of the License at
 * http://license.openmrs.org
 *
 * Software distributed under the License is distributed on an "AS IS"
 * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
 * License for the specific language governing rights and limitations
 * under the License.
 *
 * Copyright (C) OpenMRS, LLC.  All Rights Reserved.
 */

package org.openmrs.module.kenyaemr.metadata.sync;

import org.apache.commons.lang3.StringUtils;
import org.openmrs.Location;
import org.openmrs.LocationAttribute;
import org.openmrs.LocationAttributeType;

This file has been truncated. show original

github.com

I-TECH/openmrs-module-kenyaemr/blob/master/api/src/main/java/org/openmrs/module/kenyaemr/metadata/sync/LocationMflSynchronization.java

/**
 * The contents of this file are subject to the OpenMRS Public License
 * Version 1.0 (the "License"); you may not use this file except in
 * compliance with the License. You may obtain a copy of the License at
 * http://license.openmrs.org
 *
 * Software distributed under the License is distributed on an "AS IS"
 * basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the
 * License for the specific language governing rights and limitations
 * under the License.
 *
 * Copyright (C) OpenMRS, LLC.  All Rights Reserved.
 */

package org.openmrs.module.kenyaemr.metadata.sync;

import org.openmrs.Location;

import org.openmrs.api.LocationService;
import org.openmrs.module.kenyaemr.wrapper.Facility;

This file has been truncated. show original

Part of the benefit of the metadatadeploy approach is that it forces someone to actually look at the metadata as they’re describing it, and this tends to lead to fixing mistakes, rather than blindly pushing them out to lots of servers. The downside is that it takes some time.

The dataexchange module allows you to blindly export from a database, and push into other databases. If you insist that it’s necessary to push out concepts that fail OpenMRS validation, this is probably your best choice.

That said, pushing out concepts that fail validation is a bad idea and I recommend strongly against it. It’s only going to cause you more pain in the long run.

ssmusoke · February 4, 2016, 7:24am

@raff I have gone the route that you and @darius proposed using metadata sharing rather than a blind export of the concept dictionary.

I have hit a blocker details https://gist.github.com/ssmusoke/b2ff614981fb3d02fc43

There is an error exporting the concept name tags - I thought it was a data issue, but fixed the concept_name_tag_map table by removing duplicates
There seems to be a Concept 126, that is being searched for which does not exist in the database, but the export task does not state where this concept is being used for

Thanks in advance

raff · February 4, 2016, 9:23am

Seems like a bug in ConceptNameTagValidator in openmrs-core. What openmrs version are you using? Is it possible for you to upgrade to SNAPSHOT if we fix it in core?
The bad row is hard to track down based on logs indeed. We need to add more logging in metadatasharing to better indicate what is being validated. Let me address that right away.

raff · February 4, 2016, 9:41am

2nd is addressed in https://issues.openmrs.org/browse/META-352. Please test the omod attached to issue.

ssmusoke · February 4, 2016, 11:02am

@raff I am using 1.11.5 and yes it is possible for me to use a snapshot version
The fix in the attached omod works very well, as I found the concept was in a concept set which is now fixed.

ssmusoke · February 16, 2016, 8:47am

@raff The attached omod does not have the labels resolved from the messages file, even when I build from source and install in the local maven repo

raff · February 16, 2016, 10:16am

Thanks, I’ll have a look…