Travis CI failure after upgrading lucene sub Libraries

@jwnasambu Well, the underlying issue is reported here in the logs:

Caused by: org.hibernate.cache.CacheException: net.sf.ehcache.CacheException: Another CacheManager with same name 'hibernateCache' already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following:

1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary

2. Shutdown the earlier cacheManager before creating new one with same name.

The source of the existing CacheManager is: DefaultConfigurationSource [ ehcache.xml or ehcache-failsafe.xml ]

Unfortunately, without attaching a debugger, it’s a little difficult to guess why a second cache instance is suddenly being created. I’ll get back to you when I have some ideas on a way forward.

Just wondering if you can guide me as I continue digging deeper in lucene to solve this issue.

@ibacher I realized that @dkayiwa upgraded Spring and Hibernate in OpenMRS platform which gave me a moral to revisit this ticket but still after making the changes from the current release which is 8.5.0 to the least which is 5.0.0. the build still fails locally. I stand to be corrected but I want to believe that upgrading the jdk to a version that is compatible with Lucene 5 sub libraries may solve this problem since lucene not only depend on Hibernate and spring but also jdk. Java 14 would have been the best option but it seems it would be short lived since java 15 is to be released in September 2020. Could you be having an idea on how I can go about this?

Hi @jwnasambu!

So when I upgrade the Lucene version to 8.5.0 and run mvn clean package I get compiler errors like this:

[ERROR] /Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/LuceneQuery.java:[21,32] error: cannot find symbol

[ERROR]  package org.apache.lucene.queries
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/LuceneQuery.java:[49,9] error: cannot find symbol

[ERROR]
    T extends Object declared in class LuceneQuery
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[18,31] error: cannot find symbol

[ERROR]  package org.apache.lucene.search
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[20,31] error: cannot find symbol

[ERROR]  package org.apache.lucene.search
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[51,8] error: cannot find symbol

These actually seem to be errors related to the new Lucene version and API changes (OpenMRS was previously using Lucene 4.x, so leaping to 8.5.0 is quite the leap!). Fortunately, these changes are confined to two classes: TermsFilterFactory and LuceneQuery.

The actual change causing this error seems to have been made in Lucene 6 where we find:

Removal of Filter and FilteredQuery (LUCENE-6301,LUCENE-6583)

Filter and FilteredQuery have been removed. Regular queries can be used instead of filters as they have been optimized for the filtering case. And you can construct a BooleanQuery with one MUST clause for the query, and one FILTER clause for the filter in order to have similar behaviour to FilteredQuery.

And, indeed the part that’s misbehaving is the TermFilter class, which we should replace with the corresponding Query.

Now we hit a snag: If we look at the LuceneQuery class, we see that it’s using a FullTextQuery to query Lucene. This FullTextQuery isn’t part of the Lucene library, but of the Hibernate Search library. So we’ll need to update that too. However, when I look at the latest stable release of Hibernate Search, I see it only supports Hibernate 5.4. Since we’ve only upgraded to Hibernate 5.0, we should pick a version that supports that. The latest version that supports Hibernate 5.0 is Hibernate Search 5.6, which only supports Lucene 5.5. So let’s update to those versions.

When I do that locally, I end up with a compilation error in the DelegatingFullTextSession class, because it needs a new method, which can look something like this:

@Override
public FullTextQuery createFullTextQuery(QueryDescriptor descriptor, Class<?>... entities) {
	if (entities.length > 1) {
		throw new DAOException("Can't create FullTextQuery for multiple persistent classes");
	}

	if (log.isDebugEnabled()) {
		log.debug("Creating new FullTextQuery instance");
	}

	Class<?> entityClass = entities[0];
	FullTextQuery query = delegate.createFullTextQuery(descriptor, entityClass);

	if (log.isDebugEnabled()) {
		log.debug("Notifying FullTextQueryCreated listeners...");
	}

	//Notify listeners, note that we intentionally don't catch any exception from a listener
	//so that failure should just halt the entire creation operation, this is possible because 
	//the default ApplicationEventMulticaster in spring fires events serially in the same thread
	//but has the downside of where a rogue listener can block the entire application.
	FullTextQueryAndEntityClass queryAndClass = new FullTextQueryAndEntityClass(query, entityClass);
	eventPublisher.publishEvent(new FullTextQueryCreatedEvent(queryAndClass));

	return query;
}

At this point, things compile, so we should hopefully be good. Let me know if you have any questions or if you come across any more issues with this. If not, you should open some tickets to get rid of our dependence on the deprecated TermFilter class.

1 Like

Thanks for your prompt response. Let me try to digest by working it out basing on your advice then share the feedback after trying out.

@ibacher I followed all the steps as suggested above but ended up with a compilation error in the DelegatingFullTextSession class as you had stated prior. You suggested the need of a new method to solve the problem but I realized the methods already exists. that leave me with an option of open some tickets to get rid of our dependence on the deprecated TermFilter class but still figuring out the would be effect at the end.

@jwnasambu So there’s a very similar method to the one I proposed. The difference is that that method takes a Lucene Query object rather than a Lucene QueryDefinition object. Unfortunately, there are no super-classes in common between Query and QueryDefinition, so we’re stuck with two implementations.

In terms of TermFilter, we seem to be using that to include or exclude certain words from a query. I think the goal would be to replace the TermFilter with a TermQuery. This isn’t just a find-and-replace though as some of the classes will likely need to be completely rewritten, though most of the relevant work seems to already be done in the TermsFilterFactory class. That class simply returns a Filter rather than a BooleanQuery.

Even better would be to re-write the query generation so that it used Hibernate Search’s SearchFactory#buildQueryBuilder() to create the FullTextQuery. I hope that’s clear enough?

Am trying it out.

Is there a short cut to reduce the building process after making changes on the core?

You might try running mvn package instead of mvn clean package, but just pushing a change to a branch and letting Travis do it’s work may be faster for you.

Thanks sir.

@ibacher kindly am dropping off this ticket. Am so demotivated I feel I need to re-organize myself. Sorry for not fulfilling the promise of working on it to the end.

@ibacher am back to work now atleast the measles rashes are contained.

1 Like

It is perfectly fine to un assign yourself from a ticket if you do not seem to make as much progress as you would love. Some times it could simply be that the complexity was underestimated (which is also normal and happens to all of us) by whoever created the ticket.

Therefore, instead of being demotivated, feel that you were strong and courageous enough to give it a try and go this far. This is a worthwhile experience and has built your resilience for the next tickets that you are going to work on. You have stimulated a discussion that has provided a good base for whoever will take over. That is what we call team work. We win together.

So, if you can add a link to this talk thread on the ticket, whoever will take it to the finishing line, will always be grateful for the journey you started. :running_woman:

1 Like

Thanks for the advice and I really appreciate your encouragement only that I decided to give it another shot when I saw no one has claimed the ticket. I changed the approach this time and broke it into small tasks to handle. Hopefully by the end of the end I will share the out come.

@ibacher I have managed to make some changes as per your guide and here is the pull request https://github.com/openmrs/openmrs-core/pull/3174. Kindly feel free to comment on the way forward. Mean while, am looking at the cause and solution to the branch conflicts.

Hi @jwnasambu! I added a comment explaining the origin of the commits. Looks like @dkayiwa built off your work here. I’ve gone ahead and merged the other changes and I think we’re (finally) all set with this issue. Thank you for your perseverance with this… upgrading core libraries is hard work!

@ibacher Thanks very much I have seen the comment I had not fixed the link to the repository on my ticket because I was waited for the changes to be fixed.

@ibacher am happy it has come to pass and I have learned something new in the process . @dkayiwa I wanted to cling on because it helps me grow than running to comfort zone. promise you, I will never ignore any step in code it has been a nice lesson.