I will be using this thread for the next few months to put my daily updates and progress on the project.
Project: Export and Import feature for the OMRS reporting module
Project Proposal: Link
Previous thread I used for the requirement gathering: GSOC 2021 - Export and import feature for the OMRS reporting module: Requirement gathering
Mentors
Attendees
Notes
report accessibility is stored in the database https://github.com/openmrs/openmrs-module-reporting/blob/master/api/src/main/java/[ā¦]nmrs/module/reporting/report/service/db/HibernateReportDAO.java
Tasks
2021-05-20T18:30:00Z update
Today I was trying to get a period indicator report metadata bundle out from the OpenMRS metadata sharing module but just couldnāt, the existing reports donāt show up in the list. (Screenshots attached)
Tried exporting some other metadata ā concepts, which was there, about 400 of them ā and ran into an error.
I thought it might be something wrong with the version in my OpenMRS instance and cloned the latest version from github and updated the module. But still ran into the same issues.
Should play with it a bit more and see.
Period Indicator Report Definition Metadata List (With a few created in the instance)
Exporting Concepts
Tomorrow, Iām planning to go through the metadata sharing documentation properly and figure out how to use it correctly.
cc: @k.joseph
2021-05-21T18:30:00Z Update
Today I tried to import a period indicator metadata package even though it shows 0 on the list and import it into another instance, but for some reason, the importing didnāt go well, got this big internal server error, which was a java array index error. Iād be trying to do the same thing and figure what exactly is going on in the metadata sharing module tomorrow as well.
cc: @k.joseph
Thanks a lot for the updates @heshan,
I think metadata sharing does some object validation before export/import. You will need to install the module programmatically and do some debugging to investigate easier ways to go around these.
I see, thank you for the feedback
What do you mean by installing the module programmatically? Building the omod file and using the manage modules option to import them is the method that Iām using now.
right, i mean rebuilding but rather use OpenMRS SDK - Documentation - OpenMRS Wiki to easily manage deployments of changes
Sure, Iāll give it a try, thank you
2021-05-23T18:30:00Z Update
Sorry, I couldnāt put an update yesterday, I accidentally ran an rm -rf command in the root dir and ran into some issues, had to reinstall MySQL, java, and maven.
So today figured out how to use the metadata sharing module properly to export reports and other metadata like cohort queries. They were not listed as periodIndictorReportDefinitions or Cohorts but as serializable objects in the export menu. Anyhow, I got it to work, even the import function works fine. Seems like there is a mechanism to create new or merge the metadata if itās already exiting in the instance.
The error I posted two days back was an array index out of bound error which I suspect was the result of me trying to import an empty array of metadata.
Tomorrow Iāll be trying to find out how it would work if there is a matched metadata set in the system when importing.
Thanks for the tip on rebuilding and deploying modules, sure it came in handy.
cc: @k.joseph
2021-05-24T18:30:00Z Update
Today my goal was to find out how the conflicts are managed when importing objects into the module. In the module, there is a way for us to assess the objects we select before importing them. If there is a metadata object with the same UUID we can set it to merge with the existing object, this falls under the exact match category. If there is a possible match, such as metadata with the same name but different UUIDs even that is getting properly identified and the module gives us the option to choose what to do with it. For tomorrows work now that I know how the module works I think I can get into the codebase and see whatās happening under the hood.
cc: @k.joseph
Attendees
notes
@heshan updates:
Metadata sharing module and dhis connector module are in two different repositories @heshan was having doubts about how to integrate metadata sharing module functions into the connector module.
@k.joseph explained how it can be tackled using maven by setting the metadata sharing module as a dependency for the connector module
@k.joseph briefly explained the project structure of the metadata sharing module and suggested studying the OpenMRS architecture.
What should do if there are two reports with the same name(possible match)? @k.joseph suggested only rely on the UUIDs
Tasks for the next week
Shared Links
https://wiki.openmrs.org/pages/viewpage.action?pageId=22380555&navigatingVersions=true https://wiki.openmrs.org/display/docs/Technical+Details+for+Metadata+Sharing+Module
http://devmanual.openmrs.org/en/Technology/architecture.html http://devmanual.openmrs.org/en/Case_study/yourFirstModule.html
2021-05-27T18:30:00Z Update
Today I wrote my first blog post and made a short video on the metadata sharing module. Tomorrow Iāll be looking into the import issue on dhis connector module.
2021-05-28T18:30:00Z Update
Today, as I had planned, I looked into the dhis mapping import issue and figured out in what cases and why itās occurring.
The issue: When importing dhis mappings into a new instance it doesnāt do anything to import the files or show up an error but just goes on to refresh the page.
This issue occurs whenever a user is trying to import a dhis mapping into an instance that has never created a dhis mapping in it. For instance, if somebody is to create a new openMRS instance and try to import a dhis mapping the import function wouldnāt work. The following error is thrown when trying to do so
The reason for this issue is that it doesnāt create, dhisconnector/mappings/, the directory where all the mappings files should be stored, in the upload mappings function but only in the function which we use to create new dhis mappings.
The solution for this is to use the built-in java File method .mkdirs() on the mappings directory before uploading the mapping file
@k.joseph I tested out this solution and itās working fine. Should I send a PR with the change that Iāve done here? but first Iād have to create an issue on Jira under dhis connector module project right?
2021-05-29T18:30:00Z Update
I created a Jira Issue and sent a PR with the fix for the above issue. https://issues.openmrs.org/browse/DCM-35
Iām planning to study the openmrs architecture tomorrow.
cc: @k.joseph
2021-05-30T18:30:00Z Update
I tried to make metadatasharing-api-common a dependency of dhisconnector by marking it as a one in the pom.xml file today and it seems to work fine. The next step is to study the metadata sharing module export and import functions to figure out what services and how I should use them. Was on that the entire evening, Will be doing it tomorrow as well.
2021-05-31T18:30:00Z Update
Iāve been studying the metadata sharing and dhis connector module both today and been doing some experiments calling MetadataSharingServiceImpl functions from the dhis module.
Here is a pseudo function I wrote to import dhis mappings with all the underlying metadata.
MetaDataSharingSerivceImpl metadataSharingSerivce;
public void import (MultipartFile bundle) {
if (bundle.getOriginalName().endsWith(".zip")) {
File tempBundle = new File(tempFolder + bundle.getOriginalName());
bundle.transferTo(tempBundle);
ZipFile zipFile = new ZipFile(tempBundle);
List<File> fileList = unzip(zipFile);
for (File f: fileList) {
if (f.getName().endsWith(MAPPING_SUFFIX)) {
File newMapping = new File(mappingDirectory + f.getName());
f.transferTo(newMapping);
}
if (f.getName().endsWith(PERIOD_INDICATOR_REPORT_SUFFIX)) {
ObjectMapper mapprt = new ObjectMapper();
PeriodIndicatorReportDefinition pird = mapper
.readValue(f, PeriodIndicatorReportDefinition.class);
metadataSharingService.saveItem(PeriodIndicatorReportDefinition.class, pird);
}
if (f.getName().endsWith(COHORT_SUFFIX)) {
// Same procedure as above
}
// Continue for all the metadata types
}
} else {
// print an error message accordingly
}
}
cc: @k.joseph any thoughts??
Wonderful, You can go ahead and test that out, wrap it into a PR to invite more reviews.
Sure, I will try that tomorrow 
2021-06-02T18:30:00Z Update
Here is the pseudo function for the export part. This is not accurate by any means but just to organize my thoughts on how to implement these functions.
MetadataSharingServiceImpl metadataSharingService = new MetadataSharingServiceImpl();
Public String[] export (String mapping) {
String sourceDirectory = OpenmrsUtil.getApplicationDataDirectory() + DHIS_CONNECTOR_MAPPINGS_FOLDER + file.separator;
String tempDirectory = OpenmrsUtil.getApplicationDataDirectory() + DHIS_CONNECTOR_TEMP_FOLDER + file.separator;
ObjectMapper mapper = new ObjectMapper();
String msg = āā;
String path = āā;
String[] output = new String[2];
File mappingDirectory = new File(sourceDirectory);
File[] mappingFiles = mappingDirectory.listFiles();
if (mappingFiles.includes(mapping)) {
List<File> fileList = new ArrayList<>();
File mappingFile = new File(sourceDirectory + mapping);
DHISMapping mapping = mapper.readValue(mappingFile, DHISMapping.class);
String reportGUID = mapping.getPeriodIndicatorReportGUID();
PeriodIndicatorReportDefinition pird = metadataSharingService.getItem(PeriodIndicatorReportDefinition.class, reportGUID);
File pirdFile = new File(tempDirectory + pird.getName() + PERIOD_INDICATOR_REPORT_SUFFIX);
mapper.writeValue(pirdFile, pird);
// get Cohort file from pird
// get DataSetDefinition from pird
fileList.add(mappingFile);
fileList.add(pirdFile);
fileList.add(cohortFile);
fileList.add(dataSetDefinition);
path = zipMappingBundle(tempDirectory, fileList);
msg = āSuccessfully bundled the metadata with the mappingā;
} else {
msg = āThe Requested Mapping Doesnāt Existā;
}
output[0] = msg;
output[1] = path;
return output;
}
2021-06-03T18:30:00Z Update
Iāve been trying to implement the exporting feature the entire today but kept getting this error and I couldnāt move forward.
Let me explain what I have done here, where this error is coming from. Iām trying to get the UUID of the period indicator report from the mapping file and then ask MetadataServiceās get item method to get the periodIndicatorReportDefintion object.
and to get the MetadataService available in the DHIS connector module I imported the metadatasharing-api by including it as a dependency in the pom.xml files.
pom.xml (dhisconnector)
pom.xml (dhisconnector-api)
but when I import and invoke the MetadataService it puts out the above error saying the service is not found,
Do I need to do something additional to just putting the module as a dependancy? I tried to build and put in the metadata sharing module by myself but even that didnāt work.
@k.joseph can you spot anything Iām missing here?
