Lucene Search index rebuild

Tags: #<Tag:0x00007f56a3139218>

We are testing upgrading to 2.1.2 at Ampath but I have noticed it takes a lot of time updating the search index on the first run. Like more than 15 hours on the testing machine. Which make the 'it may take a few minutes ’ message misleading. During this time the system can’t start. I believe it calls https://github.com/openmrs/openmrs-core/blob/master/api/src/main/java/org/openmrs/api/db/hibernate/HibernateContextDAO.java#L500. But am curious about this https://github.com/openmrs/openmrs-core/blob/master/api/src/main/java/org/openmrs/api/db/hibernate/HibernateContextDAO.java#L522. So my question is can this be made asynchronous and allow the system to at least start? Am concerned that if something happens and you were to reindex you will take the system down for a very long time.

How many concepts do you have in your db? It’s unusual for the indexing to take that long. Maybe you are missing some db indexes. Please run https://dev.mysql.com/doc/refman/5.7/en/show-index.html for all concept tables i.e concept, concept_name, concept_datatatype, etc. and compare with a clean install of openmrs 2.1.2.

The operation is done synchronously when starting up the server, but then when you have the server running and want to rebuild an index through the administration page, it is asynchronous and you can continue to use the system.

@raff is there any time that the index gets rebuild automatically besides initial startup? We have some smoke tests that insert (and then remove) patients directly into the database and so it’s looking like we need to modify these tests to trigger a rebuild after both the insert and the removal.

Take care, Mark

Also, to @achachiez’s question… could it be an issue with the number of patients in their system? We have 300,000+ patients in one of our implementations, and I’m curious (and slightly worried :slight_smile: ) to see how long it takes to build that index.

Take care, Mark

@mogoodrich @raff We have 7500+ concepts, 840,000+ patients, 1,150,000+ patient identifiers, 850,000+ person names. I believe this are some of the tables being indexed. We moved openmrs to a dedicated server and made a few tweaks to the method doing the the indexing to use more resources and now it took like 4 hours to index and start for the first time. After that things should be smooth. The patient search is faster now.

@mogoodrich, it happens automatically on initial startup or may happen for some openmrs-core upgrades, which will require re-indexing.

The index is updated for each insert/update/delete by Hibernate after committing to the DB. Since our tests are setup in a way that there are no commits to the DB, you need to trigger updating the index manually by calling super.updateSearchIndex() to see changes in the index, see e.g. here.

@achachiez, what tweaks in particular did you do to the indexing method?

@Override
public void updateSearchIndex() {
	try {
		log.info("Updating the search index... It may take a few minutes.");
		Search.getFullTextSession(sessionFactory.getCurrentSession()).createIndexer().typesToIndexInParallel(4).batchSizeToLoadObjects(25)
				.cacheMode(CacheMode.NORMAL).threadsToLoadObjects(10).idFetchSize(150).startAndWait();
		GlobalProperty gp = Context.getAdministrationService()
				.getGlobalPropertyObject(OpenmrsConstants.GP_SEARCH_INDEX_VERSION);
		if (gp == null) {
			gp = new GlobalProperty(OpenmrsConstants.GP_SEARCH_INDEX_VERSION);
		}
		gp.setPropertyValue(OpenmrsConstants.SEARCH_INDEX_VERSION.toString());
		Context.getAdministrationService().saveGlobalProperty(gp);
		log.info("Finished updating the search index");
	} catch (Exception e) {
		throw new RuntimeException("Failed to update the search index", e);
	}
}

@raff,

So when you say “initial startup” you just mean a one-time thing, the first time you start up after installing 2.1.x? It doesn’t appear to happen at every startup. This does make sense to me… no point in rebuilding it every time, I’ll just need to get use to remembering that fact when/if I’m doing any direct database manipulation behind the scenes.

The tricky thing in our case was it was Selenium tests (running off a remote machine) that were directly inserting data into the database, so I didn’t have access to the API… I just modified the tests to navigate to the admin page and click the “refresh index” button to get them to work…

Also, I need to look into it more, but we may want to refactor the core apps patient search widget after this… I think it may make multiple search calls, “fast” identifier-only ones, and “slower” name-matching ones that may be redundant under Lucene… I plan to look into this more and ticket if needed (and if one hasn’t been created already).

Take care, Mark

Correct.

We should implement a proper REST endpoint in webservices.rest for triggering reindexing and checking the status.

Yes, it should be fast enough to do it in one call.

Thanks for sharing your settings! In addition to that did you tune any Hibernate settings or you used OpenMRS defaults?

We should look into finding the most reasonable defaults that will work for most implementations.

Created TRUNK-5329. @mogoodrich, I’ll appreciate your input as well after running it for your instances.

I tried hitting the REST endpoint that the button uses, but it seemed like Selenium didn’t have a built-in way to hit an endpoint, so I used httpclient, but, unsurprising that failed because it wasn’t authenicated. At that point I figured it was easier just to use Selenium to hit the button… :slight_smile:

@raff I used the default settings.

Hello,

I’d like to know if for the first startup, is it possible to perform the search index in background ? Otherwise, the system is not accessible for 1 hour in our case …

For information, I tried to launch a performance analysis with mysql workbench during the index job and I noticed there are several full scans done on some concepts tables ( will try to send a snapshot). Thanks Fred

How did you confirm that the inaccessibility was caused by building the search index?

it seems clear from the log and this line: Updating the search index... It may take a few minutes. and this message from @raff:

The operation is done synchronously when starting up the server, but then when you have the server running and want to rebuild an index through the administration page, it is asynchronous and you can continue to use the system.

And I tried to access to the application before the end of the indexation without any success.

Which version of OpenMRS are you running?

We use version 2.3.0

Do you mean OpenMRS Reference Application version 2.3?

sorry… reference application version is 2.10.0-SNAPSHOT

Yes it is possible by using: https://github.com/openmrs/openmrs-core/blob/2.3.0/api/src/main/java/org/openmrs/api/db/hibernate/HibernateContextDAO.java#L554

Instead of: https://github.com/openmrs/openmrs-core/blob/2.3.0/api/src/main/java/org/openmrs/api/db/hibernate/HibernateContextDAO.java#L532