Export concepts using data exchange to preserve IDs for later updates of CIEL using mdsbuilder

Development
1

Hi everyone,

I was following the steps for releasing the Reference Metadata Module, unfortunately, when I try to export the concepts as mentioned here , I am greeted with a failure. Here are the logs for the same.

Any help would be appreciated!

cc: @dkayiwa , @raff , @mogoodrich

2

After looking at this commit: https://github.com/openmrs/openmrs-module-dataexchange/commit/3a17c7f77d92f3f33c1ee4320ebb5599fa704106#diff-0016cc7be122bba9ec3fda755c6369ac and this line: https://github.com/openmrs/openmrs-module-dataexchange/blob/469535f8a33b09e432e0eca24d78774bb09b0172/omod/src/main/java/org/openmrs/module/dataexchange/web/controller/MetadataSharingParser.java#L12 i conclude by saying that our documentation is pointing to a feature which was never implemented. @raff am i correct? :slight_smile:

3

I’ve definitely implemented that at some point. Maybe I haven’t pushed changes to the repo or they were overwritten… Strange that mdsbuilder doesn’t have them as well. Again I need to look into my backups to see, if I can recover anything.

4

@raff did you by any chance recover this code? :slight_smile:

5

@dkayiwa is there a manual way to do this? As in, before the implementation of this feature , I guess there would’ve been a way to do this. So maybe I could go with that temporarily untill @raff finds the code?

6

@dkayiwa, not yet. I keep my backups on a different machine and will have access to that in the evening.

@reubenv, there’s no other way. We used that approach since the beginning of RA, but somehow code was lost in the repo and not properly released (my bad).

7

I didn’t find any backups of dataexchange, which means we lost the code for exporting concepts included in MDS packages unless someone else has it. The good thing is it’s fairly simple to rewrite. Before doing that, I’d recommend installing locally dataexchange 1.3.2, 1.3.1, 1.3, 1.2 to see, if any of those versions work. It’s possible that a working version was released at some point.

8

@raff when i look at the commit history https://github.com/openmrs/openmrs-module-dataexchange/commits/master this feature was added and released in version 1.3 but with a MetadataSharingParser class whose parseConceptIds method body just throws a not supported exception, from the first time of its release up to the latest release.

Do you happen to know why in all subsequent releases the MetadataSharingParser class just throws that exception?

9

It was a placeholder before implementing it, which I did later and also deployed to mdsbuilder an omod, but apparently I failed to push changes to upstream and do a proper release to maven. Now after restoring mdsbuilder we no longer have even the binary.

10

@raff in your estimation, how long do you think it would take some one to reimplement this feature?

11

I’d say < 2h. It’s a matter of parsing header.xml from a zipped metadata package to extract concepts ids. The rest is done in https://github.com/openmrs/openmrs-module-dataexchange/blob/master/omod/src/main/java/org/openmrs/module/dataexchange/web/controller/DataExchangeController.java#L69

12

Hi @raff @dkayiwa ,

Any updates on the feature for the data-exchange module?

13

I may have time to implement that by the end of this week.

14

So, my understanding is RefApp 2.9 release is blocked by the need to implement parseConceptIds(InputStream) method to

  • treat input stream as zip input stream
    ZipInputStream zis = new ZipInputStream(fin);
  • find and read header.xml contents into memory
  • parse as XML document
  • return values from the following xpath(s) into a Set:
    • /package/items/org.openmrs.module.metadatasharing.Item[classname='org.openmrs.Concept']/id
    • /package/relatedItems/org.openmrs.module.metadatasharing.Item[classname='org.openmrs.Concept']/id
  • close the input stream before returning (to clean up)

Am I close? Is it safe to assume the header.xml can be loaded into memory?

15

Burke, yes, that’s it and it’s safe to load header.xml into memory.

16

Committed in https://github.com/openmrs/openmrs-module-dataexchange/commit/39f6ec49bbfd930a2aa8cb787356954c2b224aa3

Now it’s a matter of releasing the module, testing and deploying to mdsbuilder. I’m not setup to do a release on my machine so please do that and I can update mdsbuilder to include the fixed version.

17

Awesome @raff! I’ll try releasing the module now(if I have the necessary permissions) :smile:

18

@dkayiwa , do we have a CI plan for this module? I am not really sure about the procedure for the releasing this particular one.

19

I have created the CI plan and released version 1.3.3 of the module: https://ci.openmrs.org/browse/DEX-DEX-3

20

@dkayiwa , you’re awesome! :smiley: , @raff can you update mdsbuilder please?( I have a feeling that the updation part is something that I can also do but I don’t want to take chances here ). Also , should we be updating the database on mds with the latest CIEL dump file which was released just 2-3 days back?