Travis CI failure after upgrading lucene sub Libraries

My friend, am still working on it that is why I asked for help. I will only surrender when am done.

:grinning: dont surrender my friend take your time, @ibacher information will push you somewhere i know

I have tried bumping Hibernate Search version to 5.5.8.Final and the Lucene version to 5.3.1and this is the error log https://pastebin.com/mj99K9f5. Building a core needs patience of a saint. It has been taking me at least 7hrs. Is there away of reducing on time spend running the core after making changes?

Hmmmm… 7 hours is excessive. Some thoughts on what you can do:

  1. Continue using Travis and just push the changes you’re testing to a PR (it might be better if you used a draft PR when doing this)

  2. Instead of mvn clean install run mvn -pl :api -am clean install. This will only build the api module, which is still quite heavy, but might build quicker in your environment.

Looking at the build log, I recommend option 1. Could you zip up the files in: C:\Users\JULIE\OpenMRS-Tickets\openmrs-core\api\target\surefire-reports and attach that file here?

The Travis CI is still falling as per the PR https://github.com/openmrs/openmrs-core/pull/3121. I tried to zip and share the file surefire-reports but I failed because of the unauthorized format of the content.

Please use pastebin for your log.

You are right. I tried to use paste bin but it couldn’t copy the content @ibacher requested me to zip up the files from: C:\Users\JULIE\OpenMRS-Tickets\openmrs-core\api\target\surefire-reports that is why the mess.

@dkayiwa and @ibacher how do I go about this issue. I have run out of options but I desire to complete this ticket.

@jwnasambu Well, the underlying issue is reported here in the logs:

Caused by: org.hibernate.cache.CacheException: net.sf.ehcache.CacheException: Another CacheManager with same name 'hibernateCache' already exists in the same VM. Please provide unique names for each CacheManager in the config or do one of following:

1. Use one of the CacheManager.create() static factory methods to reuse same CacheManager with same name or create one if necessary

2. Shutdown the earlier cacheManager before creating new one with same name.

The source of the existing CacheManager is: DefaultConfigurationSource [ ehcache.xml or ehcache-failsafe.xml ]

Unfortunately, without attaching a debugger, it’s a little difficult to guess why a second cache instance is suddenly being created. I’ll get back to you when I have some ideas on a way forward.

Just wondering if you can guide me as I continue digging deeper in lucene to solve this issue.

@ibacher I realized that @dkayiwa upgraded Spring and Hibernate in OpenMRS platform which gave me a moral to revisit this ticket but still after making the changes from the current release which is 8.5.0 to the least which is 5.0.0. the build still fails locally. I stand to be corrected but I want to believe that upgrading the jdk to a version that is compatible with Lucene 5 sub libraries may solve this problem since lucene not only depend on Hibernate and spring but also jdk. Java 14 would have been the best option but it seems it would be short lived since java 15 is to be released in September 2020. Could you be having an idea on how I can go about this?

Hi @jwnasambu!

So when I upgrade the Lucene version to 8.5.0 and run mvn clean package I get compiler errors like this:

[ERROR] /Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/LuceneQuery.java:[21,32] error: cannot find symbol

[ERROR]  package org.apache.lucene.queries
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/LuceneQuery.java:[49,9] error: cannot find symbol

[ERROR]
    T extends Object declared in class LuceneQuery
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[18,31] error: cannot find symbol

[ERROR]  package org.apache.lucene.search
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[20,31] error: cannot find symbol

[ERROR]  package org.apache.lucene.search
/Users/ibacher/Documents/openmrs/openmrs-core/api/src/main/java/org/openmrs/api/db/hibernate/search/TermsFilterFactory.java:[51,8] error: cannot find symbol

These actually seem to be errors related to the new Lucene version and API changes (OpenMRS was previously using Lucene 4.x, so leaping to 8.5.0 is quite the leap!). Fortunately, these changes are confined to two classes: TermsFilterFactory and LuceneQuery.

The actual change causing this error seems to have been made in Lucene 6 where we find:

Removal of Filter and FilteredQuery (LUCENE-6301,LUCENE-6583)

Filter and FilteredQuery have been removed. Regular queries can be used instead of filters as they have been optimized for the filtering case. And you can construct a BooleanQuery with one MUST clause for the query, and one FILTER clause for the filter in order to have similar behaviour to FilteredQuery.

And, indeed the part that’s misbehaving is the TermFilter class, which we should replace with the corresponding Query.

Now we hit a snag: If we look at the LuceneQuery class, we see that it’s using a FullTextQuery to query Lucene. This FullTextQuery isn’t part of the Lucene library, but of the Hibernate Search library. So we’ll need to update that too. However, when I look at the latest stable release of Hibernate Search, I see it only supports Hibernate 5.4. Since we’ve only upgraded to Hibernate 5.0, we should pick a version that supports that. The latest version that supports Hibernate 5.0 is Hibernate Search 5.6, which only supports Lucene 5.5. So let’s update to those versions.

When I do that locally, I end up with a compilation error in the DelegatingFullTextSession class, because it needs a new method, which can look something like this:

@Override
public FullTextQuery createFullTextQuery(QueryDescriptor descriptor, Class<?>... entities) {
	if (entities.length > 1) {
		throw new DAOException("Can't create FullTextQuery for multiple persistent classes");
	}

	if (log.isDebugEnabled()) {
		log.debug("Creating new FullTextQuery instance");
	}

	Class<?> entityClass = entities[0];
	FullTextQuery query = delegate.createFullTextQuery(descriptor, entityClass);

	if (log.isDebugEnabled()) {
		log.debug("Notifying FullTextQueryCreated listeners...");
	}

	//Notify listeners, note that we intentionally don't catch any exception from a listener
	//so that failure should just halt the entire creation operation, this is possible because 
	//the default ApplicationEventMulticaster in spring fires events serially in the same thread
	//but has the downside of where a rogue listener can block the entire application.
	FullTextQueryAndEntityClass queryAndClass = new FullTextQueryAndEntityClass(query, entityClass);
	eventPublisher.publishEvent(new FullTextQueryCreatedEvent(queryAndClass));

	return query;
}

At this point, things compile, so we should hopefully be good. Let me know if you have any questions or if you come across any more issues with this. If not, you should open some tickets to get rid of our dependence on the deprecated TermFilter class.

1 Like

Thanks for your prompt response. Let me try to digest by working it out basing on your advice then share the feedback after trying out.

@ibacher I followed all the steps as suggested above but ended up with a compilation error in the DelegatingFullTextSession class as you had stated prior. You suggested the need of a new method to solve the problem but I realized the methods already exists. that leave me with an option of open some tickets to get rid of our dependence on the deprecated TermFilter class but still figuring out the would be effect at the end.

@jwnasambu So there’s a very similar method to the one I proposed. The difference is that that method takes a Lucene Query object rather than a Lucene QueryDefinition object. Unfortunately, there are no super-classes in common between Query and QueryDefinition, so we’re stuck with two implementations.

In terms of TermFilter, we seem to be using that to include or exclude certain words from a query. I think the goal would be to replace the TermFilter with a TermQuery. This isn’t just a find-and-replace though as some of the classes will likely need to be completely rewritten, though most of the relevant work seems to already be done in the TermsFilterFactory class. That class simply returns a Filter rather than a BooleanQuery.

Even better would be to re-write the query generation so that it used Hibernate Search’s SearchFactory#buildQueryBuilder() to create the FullTextQuery. I hope that’s clear enough?

Am trying it out.

Is there a short cut to reduce the building process after making changes on the core?

You might try running mvn package instead of mvn clean package, but just pushing a change to a branch and letting Travis do it’s work may be faster for you.

Thanks sir.