Metadata Sharing Performance


As our metadata packages have grown, the time to import them has grown as well–to the point that some of our largest packages can take 15-20 minutes to load.

I’ve started poking around in the MDS code to see if I can find any slow points in the process in hopes of finding some quick wins, but I’m not super hopefully, it may just be a slow process.

However, it seems like a large portion of the time is taken up in the “resolveRelatedImportItems” and in particular this for loop:

No worries since it’s been like 5 years now, but do you happen remember what this block of code does? I’m starting to dive into now, but it seemed worth shooting you a quick note in case it jogged your memory at all as to why this may be slow. I’m going to try to take a look with a profiler.

Thanks & take care, Mark

In that loop I’m adding all related (implicit) import items for the items explicitly added to the package e.g. implicit answer concepts of an explicitly added question concept. The list is later used to determine, if you already have those related items and they need to be used/merged or created.

It may be worth to save the list at the export time to the header.xml, so that we would not have to traverse the dependencies again while doing actual imports. That is worth doing if most of the time is spent while traversing in, but not in, which we will need to do anyway.

Thanks @raff, this is very helpful.

It does look like the hotspots are in the getItemByUuid method, and specifically in PreparedStatement.executeQuery, so it’s really the fetching of the existing data that takes all the time, not resolving the related items. I am going to continue to explore, will let you know what I find.

Take care, Mark