GSoC 2021 - Export and Import feature for the OMRS reporting module: Updates

I will be using this thread for the next few months to put my daily updates and progress on the project.

1 Like

1st Weekly Meeting - 2021-05-19T18:30:00Z

Attendees

@heshan @k.joseph

Notes

  • @heshan walked @k.joseph through the proposal and explained the thought process and the provisional design of the solution
  • @k.joseph suggested instead of letting the user choose and import/export the report definitions and corresponding metadata alongside the DHIS mappings just export one bundle with everything needed for the mapping to work. When importing the system should be able to identify the files and make the metadata accordingly
  • @k.joseph emphasised the importance of handling the conflicts if the same metadata is already present in the system when importing.
  • @heshan had a problem finding the place where the period indictor reports are stored ultimately and @k.joseph pointed to the place where those can be found

openmrs-module-reporting/ReportServiceImpl.java at master · openmrs/openmrs-module-reporting · GitHub

report accessibility is stored in the database https://github.com/openmrs/openmrs-module-reporting/blob/master/api/src/main/java/[…]nmrs/module/reporting/report/service/db/HibernateReportDAO.java

  • @k.joseph suggested playing with the metadata sharing module a bit in this week to figure out how the metadata are exported.

Tasks

  • Get familiarized with the functionality of metadata sharing module
  • Go through the source code of metadata sharing module
  • Find if the conflicts are addressed in the exisiting module
1 Like

2021-05-20T18:30:00Z update

Today I was trying to get a period indicator report metadata bundle out from the OpenMRS metadata sharing module but just couldn’t, the existing reports don’t show up in the list. (Screenshots attached)

Tried exporting some other metadata – concepts, which was there, about 400 of them – and ran into an error.

I thought it might be something wrong with the version in my OpenMRS instance and cloned the latest version from github and updated the module. But still ran into the same issues.

Should play with it a bit more and see.

Period Indicator Report Definition Metadata List (With a few created in the instance)

Exporting Concepts

Tomorrow, I’m planning to go through the metadata sharing documentation properly and figure out how to use it correctly.

cc: @k.joseph

2021-05-21T18:30:00Z Update

Today I tried to import a period indicator metadata package even though it shows 0 on the list and import it into another instance, but for some reason, the importing didn’t go well, got this big internal server error, which was a java array index error. I’d be trying to do the same thing and figure what exactly is going on in the metadata sharing module tomorrow as well.

cc: @k.joseph

1 Like

Thanks a lot for the updates @heshan,

I think metadata sharing does some object validation before export/import. You will need to install the module programmatically and do some debugging to investigate easier ways to go around these.

1 Like

I see, thank you for the feedback
What do you mean by installing the module programmatically? Building the omod file and using the manage modules option to import them is the method that I’m using now.

right, i mean rebuilding but rather use OpenMRS SDK - Documentation - OpenMRS Wiki to easily manage deployments of changes

1 Like

Sure, I’ll give it a try, thank you

2021-05-23T18:30:00Z Update

Sorry, I couldn’t put an update yesterday, I accidentally ran an rm -rf command in the root dir and ran into some issues, had to reinstall MySQL, java, and maven.

So today figured out how to use the metadata sharing module properly to export reports and other metadata like cohort queries. They were not listed as periodIndictorReportDefinitions or Cohorts but as serializable objects in the export menu. Anyhow, I got it to work, even the import function works fine. Seems like there is a mechanism to create new or merge the metadata if it’s already exiting in the instance.

The error I posted two days back was an array index out of bound error which I suspect was the result of me trying to import an empty array of metadata.

Tomorrow I’ll be trying to find out how it would work if there is a matched metadata set in the system when importing.

Thanks for the tip on rebuilding and deploying modules, sure it came in handy.

cc: @k.joseph

1 Like

2021-05-24T18:30:00Z Update

Today my goal was to find out how the conflicts are managed when importing objects into the module. In the module, there is a way for us to assess the objects we select before importing them. If there is a metadata object with the same UUID we can set it to merge with the existing object, this falls under the exact match category. If there is a possible match, such as metadata with the same name but different UUIDs even that is getting properly identified and the module gives us the option to choose what to do with it. For tomorrows work now that I know how the module works I think I can get into the codebase and see what’s happening under the hood.

cc: @k.joseph

2nd Weekly Meeting - 2021-05-26T18:30:00Z

Attendees

@heshan @k.joseph

notes

  • @heshan updates:

    • Was having problems exporting period indicator reports but now has figured out how to do so, those can be found under serialised objects list
    • Studied the import conflict handle functions
    • Tried to export dhis mapping from the connector module
    • But couldn’t import the same mapping files into a different instance, it only works when imported to the same instance, when uploaded into a new instance even it doesn’t even show an error message
    • @k.joseph suggested inspecting it using developer tools to try to find out what’s happening
  • Metadata sharing module and dhis connector module are in two different repositories @heshan was having doubts about how to integrate metadata sharing module functions into the connector module.

  • @k.joseph explained how it can be tackled using maven by setting the metadata sharing module as a dependency for the connector module

  • @k.joseph briefly explained the project structure of the metadata sharing module and suggested studying the OpenMRS architecture.

  • What should do if there are two reports with the same name(possible match)? @k.joseph suggested only rely on the UUIDs

    • If this UUID doesn’t exist in the instance create a new object
    • If not merge the object

Tasks for the next week

  • Figure out dhis mapping importing problem
  • Look into the OpenMRS architecture
  • Study how to set an OMRS module as a maven dependency

Shared Links

https://wiki.openmrs.org/pages/viewpage.action?pageId=22380555&navigatingVersions=true https://wiki.openmrs.org/display/docs/Technical+Details+for+Metadata+Sharing+Module

http://devmanual.openmrs.org/en/Technology/architecture.html http://devmanual.openmrs.org/en/Case_study/yourFirstModule.html

2021-05-27T18:30:00Z Update

Today I wrote my first blog post and made a short video on the metadata sharing module. Tomorrow I’ll be looking into the import issue on dhis connector module.

2021-05-28T18:30:00Z Update

Today, as I had planned, I looked into the dhis mapping import issue and figured out in what cases and why it’s occurring.

  • The issue: When importing dhis mappings into a new instance it doesn’t do anything to import the files or show up an error but just goes on to refresh the page.

  • This issue occurs whenever a user is trying to import a dhis mapping into an instance that has never created a dhis mapping in it. For instance, if somebody is to create a new openMRS instance and try to import a dhis mapping the import function wouldn’t work. The following error is thrown when trying to do so

  • The reason for this issue is that it doesn’t create, dhisconnector/mappings/, the directory where all the mappings files should be stored, in the upload mappings function but only in the function which we use to create new dhis mappings.

  • The solution for this is to use the built-in java File method .mkdirs() on the mappings directory before uploading the mapping file

@k.joseph I tested out this solution and it’s working fine. Should I send a PR with the change that I’ve done here? but first I’d have to create an issue on Jira under dhis connector module project right?

2021-05-29T18:30:00Z Update

I created a Jira Issue and sent a PR with the fix for the above issue. https://issues.openmrs.org/browse/DCM-35

I’m planning to study the openmrs architecture tomorrow.

cc: @k.joseph

1 Like

2021-05-30T18:30:00Z Update

I tried to make metadatasharing-api-common a dependency of dhisconnector by marking it as a one in the pom.xml file today and it seems to work fine. The next step is to study the metadata sharing module export and import functions to figure out what services and how I should use them. Was on that the entire evening, Will be doing it tomorrow as well.

2021-05-31T18:30:00Z Update

I’ve been studying the metadata sharing and dhis connector module both today and been doing some experiments calling MetadataSharingServiceImpl functions from the dhis module. Here is a pseudo function I wrote to import dhis mappings with all the underlying metadata.

MetaDataSharingSerivceImpl metadataSharingSerivce;

public void import (MultipartFile bundle) {
if (bundle.getOriginalName().endsWith(".zip")) {

  File tempBundle = new File(tempFolder + bundle.getOriginalName());
  bundle.transferTo(tempBundle);
  ZipFile zipFile = new ZipFile(tempBundle);
  List<File> fileList = unzip(zipFile);

  for (File f: fileList) {
    if (f.getName().endsWith(MAPPING_SUFFIX)) {
      File newMapping = new File(mappingDirectory + f.getName());
      f.transferTo(newMapping);
    }
    if (f.getName().endsWith(PERIOD_INDICATOR_REPORT_SUFFIX)) {
      ObjectMapper mapprt = new ObjectMapper();
      PeriodIndicatorReportDefinition pird = mapper
                        .readValue(f, PeriodIndicatorReportDefinition.class);
      metadataSharingService.saveItem(PeriodIndicatorReportDefinition.class, pird);
    }
    if (f.getName().endsWith(COHORT_SUFFIX)) {
      // Same procedure as above
    }
    // Continue for all the metadata types 
  }
} else {
// print an error message accordingly
}
}

cc: @k.joseph any thoughts??

Wonderful, You can go ahead and test that out, wrap it into a PR to invite more reviews.

1 Like

Sure, I will try that tomorrow :blush:

2021-06-02T18:30:00Z Update

Here is the pseudo function for the export part. This is not accurate by any means but just to organize my thoughts on how to implement these functions.

MetadataSharingServiceImpl metadataSharingService = new MetadataSharingServiceImpl();

Public String[] export (String mapping) {
	String sourceDirectory = OpenmrsUtil.getApplicationDataDirectory() + DHIS_CONNECTOR_MAPPINGS_FOLDER + file.separator;
	String tempDirectory = OpenmrsUtil.getApplicationDataDirectory() + DHIS_CONNECTOR_TEMP_FOLDER + file.separator;

	ObjectMapper mapper = new ObjectMapper();

	String msg = “”;
	String path = “”;
	String[] output = new String[2];

	File mappingDirectory = new File(sourceDirectory);
	File[] mappingFiles = mappingDirectory.listFiles();

	if (mappingFiles.includes(mapping)) {
		List<File> fileList = new ArrayList<>();

		File mappingFile = new File(sourceDirectory + mapping);
		DHISMapping mapping = mapper.readValue(mappingFile, DHISMapping.class);

		String reportGUID = mapping.getPeriodIndicatorReportGUID();
		PeriodIndicatorReportDefinition pird = metadataSharingService.getItem(PeriodIndicatorReportDefinition.class, reportGUID);
		File pirdFile = new File(tempDirectory + pird.getName() + PERIOD_INDICATOR_REPORT_SUFFIX);
		mapper.writeValue(pirdFile, pird);

		// get Cohort file from pird
		// get DataSetDefinition from pird

		fileList.add(mappingFile);
		fileList.add(pirdFile);
		fileList.add(cohortFile);
		fileList.add(dataSetDefinition);
		
		path = zipMappingBundle(tempDirectory, fileList);
		msg = “Successfully bundled the metadata with the mapping”;
	} else {
		msg = “The Requested Mapping Doesn’t Exist”;
	}

	output[0] = msg;
	output[1] = path;
	return output;
}

2021-06-03T18:30:00Z Update

I’ve been trying to implement the exporting feature the entire today but kept getting this error and I couldn’t move forward.

Let me explain what I have done here, where this error is coming from. I’m trying to get the UUID of the period indicator report from the mapping file and then ask MetadataService’s get item method to get the periodIndicatorReportDefintion object.

and to get the MetadataService available in the DHIS connector module I imported the metadatasharing-api by including it as a dependency in the pom.xml files.

pom.xml (dhisconnector)

pom.xml (dhisconnector-api)

but when I import and invoke the MetadataService it puts out the above error saying the service is not found,

Do I need to do something additional to just putting the module as a dependancy? I tried to build and put in the metadata sharing module by myself but even that didn’t work.

@k.joseph can you spot anything I’m missing here?