Export concepts using data exchange to preserve IDs for later updates of CIEL using mdsbuilder

(Reuben Varghese) #1

Hi everyone,

I was following the steps for releasing the Reference Metadata Module, unfortunately, when I try to export the concepts as mentioned here , I am greeted with a failure. Here are the logs for the same.

Any help would be appreciated!

cc: @dkayiwa , @raff , @mogoodrich

(Daniel Kayiwa) #2

After looking at this commit: https://github.com/openmrs/openmrs-module-dataexchange/commit/3a17c7f77d92f3f33c1ee4320ebb5599fa704106#diff-0016cc7be122bba9ec3fda755c6369ac and this line: https://github.com/openmrs/openmrs-module-dataexchange/blob/469535f8a33b09e432e0eca24d78774bb09b0172/omod/src/main/java/org/openmrs/module/dataexchange/web/controller/MetadataSharingParser.java#L12 i conclude by saying that our documentation is pointing to a feature which was never implemented. @raff am i correct? :slight_smile:

(Rafal Korytkowski) #3

I’ve definitely implemented that at some point. Maybe I haven’t pushed changes to the repo or they were overwritten… Strange that mdsbuilder doesn’t have them as well. Again I need to look into my backups to see, if I can recover anything.

(Daniel Kayiwa) #4

@raff did you by any chance recover this code? :slight_smile:

(Reuben Varghese) #5

@dkayiwa is there a manual way to do this? As in, before the implementation of this feature , I guess there would’ve been a way to do this. So maybe I could go with that temporarily untill @raff finds the code?

(Rafal Korytkowski) #6

@dkayiwa, not yet. I keep my backups on a different machine and will have access to that in the evening.

@reubenv, there’s no other way. We used that approach since the beginning of RA, but somehow code was lost in the repo and not properly released (my bad).

(Rafal Korytkowski) #7

I didn’t find any backups of dataexchange, which means we lost the code for exporting concepts included in MDS packages unless someone else has it. The good thing is it’s fairly simple to rewrite. Before doing that, I’d recommend installing locally dataexchange 1.3.2, 1.3.1, 1.3, 1.2 to see, if any of those versions work. It’s possible that a working version was released at some point.

(Daniel Kayiwa) #8

@raff when i look at the commit history https://github.com/openmrs/openmrs-module-dataexchange/commits/master this feature was added and released in version 1.3 but with a MetadataSharingParser class whose parseConceptIds method body just throws a not supported exception, from the first time of its release up to the latest release.

Do you happen to know why in all subsequent releases the MetadataSharingParser class just throws that exception?

(Rafal Korytkowski) #9

It was a placeholder before implementing it, which I did later and also deployed to mdsbuilder an omod, but apparently I failed to push changes to upstream and do a proper release to maven. Now after restoring mdsbuilder we no longer have even the binary.

(Daniel Kayiwa) #10

@raff in your estimation, how long do you think it would take some one to reimplement this feature?

(Rafal Korytkowski) #11

I’d say < 2h. It’s a matter of parsing header.xml from a zipped metadata package to extract concepts ids. The rest is done in https://github.com/openmrs/openmrs-module-dataexchange/blob/master/omod/src/main/java/org/openmrs/module/dataexchange/web/controller/DataExchangeController.java#L69

(Reuben Varghese) #12

Hi @raff @dkayiwa ,

Any updates on the feature for the data-exchange module?

(Rafal Korytkowski) #13

I may have time to implement that by the end of this week.

(Burke Mamlin) #14

So, my understanding is RefApp 2.9 release is blocked by the need to implement parseConceptIds(InputStream) method to

  • treat input stream as zip input stream
    ZipInputStream zis = new ZipInputStream(fin);
  • find and read header.xml contents into memory
  • parse as XML document
  • return values from the following xpath(s) into a Set:
    • /package/items/org.openmrs.module.metadatasharing.Item[classname='org.openmrs.Concept']/id
    • /package/relatedItems/org.openmrs.module.metadatasharing.Item[classname='org.openmrs.Concept']/id
  • close the input stream before returning (to clean up)

Am I close? Is it safe to assume the header.xml can be loaded into memory?

(Rafal Korytkowski) #15

Burke, yes, that’s it and it’s safe to load header.xml into memory.

(Rafal Korytkowski) #16

Committed in https://github.com/openmrs/openmrs-module-dataexchange/commit/39f6ec49bbfd930a2aa8cb787356954c2b224aa3

Now it’s a matter of releasing the module, testing and deploying to mdsbuilder. I’m not setup to do a release on my machine so please do that and I can update mdsbuilder to include the fixed version.

(Reuben Varghese) #17

Awesome @raff! I’ll try releasing the module now(if I have the necessary permissions) :smile:

(Reuben Varghese) #18

@dkayiwa , do we have a CI plan for this module? I am not really sure about the procedure for the releasing this particular one.

(Daniel Kayiwa) #19

I have created the CI plan and released version 1.3.3 of the module: https://ci.openmrs.org/browse/DEX-DEX-3

(Reuben Varghese) #20

@dkayiwa , you’re awesome! :smiley: , @raff can you update mdsbuilder please?( I have a feeling that the updation part is something that I can also do but I don’t want to take chances here ). Also , should we be updating the database on mds with the latest CIEL dump file which was released just 2-3 days back?