Metadata Sharing Performance

mogoodrich · June 29, 2017, 6:26pm

As our metadata packages have grown, the time to import them has grown as well–to the point that some of our largest packages can take 15-20 minutes to load.

I’ve started poking around in the MDS code to see if I can find any slow points in the process in hopes of finding some quick wins, but I’m not super hopefully, it may just be a slow process.

However, it seems like a large portion of the time is taken up in the “resolveRelatedImportItems” and in particular this for loop:

github.com

openmrs/openmrs-module-metadatasharing/blob/master/api-common/src/main/java/org/openmrs/module/metadatasharing/wrapper/impl/PackageImporterImpl.java#L213


		@SuppressWarnings("unchecked")

		List<Object> importedMetadata = MetadataSharing.getInstance().getMetadataSerializer()

		        .deserialize(xml, List.class, importedPackage.getOpenmrsVersion());

		for (Object each : importedMetadata) {

			ImportedItem importedItem = getExistingOrNewImportedItem(each);

			tmpImportItems.put(importedItem, importedItem);

		}

		

		// We need to copy importItems to a new list as we'll be adding to importItems while iterating

		List<ImportedItem> unresolvedItems = new ArrayList<ImportedItem>(tmpImportItems.keySet());

		for (ImportedItem unresolvedItem : unresolvedItems) {

			resolveRelatedImportItems(unresolvedItem, tmpImportItems);

		}

		

		MetadataSharing.getInstance().getResolverEngine()

		        .resolve(tmpImportItems.keySet(), importedPackage.getImportConfig());

		

		List<ImportedItem> resolvedItems = new ArrayList<ImportedItem>(tmpImportItems.keySet());

		

		MetadataSharing.getService().persistImportedItems(resolvedItems);

No worries since it’s been like 5 years now, but do you happen remember what this block of code does? I’m starting to dive into now, but it seemed worth shooting you a quick note in case it jogged your memory at all as to why this may be slow. I’m going to try to take a look with a profiler.

Thanks & take care, Mark

raff · July 3, 2017, 9:30am

In that loop I’m adding all related (implicit) import items for the items explicitly added to the package e.g. implicit answer concepts of an explicitly added question concept. The list is later used to determine, if you already have those related items and they need to be used/merged or created.

It may be worth to save the list at the export time to the header.xml, so that we would not have to traverse the dependencies again while doing actual imports. That is worth doing if most of the time is spent while traversing in https://github.com/openmrs/openmrs-module-metadatasharing/blob/master/api-common/src/main/java/org/openmrs/module/metadatasharing/wrapper/impl/PackageImporterImpl.java#L265-L282, but not in https://github.com/openmrs/openmrs-module-metadatasharing/blob/master/api-common/src/main/java/org/openmrs/module/metadatasharing/wrapper/impl/PackageImporterImpl.java#L301, which we will need to do anyway.

mogoodrich · July 5, 2017, 9:33pm

Thanks @raff, this is very helpful.

It does look like the hotspots are in the getItemByUuid method, and specifically in PreparedStatement.executeQuery, so it’s really the fetching of the existing data that takes all the time, not resolving the related items. I am going to continue to explore, will let you know what I find.

Take care, Mark