Optimising search concepts by concept class

Tags: #<Tag:0x00007f75221ac4f0>

ConceptService#getConceptsByClass(…) uses Lucene under the hood! I think the reason for this was to achieve reusability of this complex method that leverages Lucene. However this gets superfluous to take such a long route when we have a ConceptClass at hand. For some reasons that I’m not certain of, relying on Lucene slows down the retrieval of data in this context! This becomes so critical if the API is consumed by an HFE tag that is processed multiple times by the <repeat/> tag!

Our Experience with the <condition/> HFE tag

The <condition/> tag somewhere relies on this API… We have an HFE form that consumes this tag. However, we may want to record upto 10 conditions in an encounter, so this tag is replicated 10 times by the <repeat/> tag; meaning this API would be hit 10 times causing a significantly slowed down yield of results! On my dev machine, it takes upto 2 minutes to load such a form; a machine with lower specs might just timeout!

Well this is quite an issue and a blocker within our implementation and we thought of having Introducing an ad-hoc DAO layer to retrieve concepts by class within HFE module for now as we discover how to optimise core?

Any thoughts or earlier findings are welcome :slight_smile:

cc: @mogoodrich, @dkayiwa, @mksd

1 Like

@mogoodrich @ball I’m kind of surprised that you guys haven’t come across this when doing stuff like this:

<obs conceptId="6178" answerClasses="Diagnosis" style="autocomplete" />

Because of this basically.

If such tag is repeated this can become deadly, especially if there are many diagnoses.

Could you provide some specific examples of the queries and use case? I can’t comment on the technical details, but I would like to understand what your form is trying to do… thanks!

@mksd it looks to me from the code you linked to that this is only an issue if it is not configured as “autocomplete”. If you use style=“autocomplete” does it still load every Concept by class each time it hits the tag, or is the autocomplete more of a dynamic search?

@mseaton is that why you might never have noticed on your end that ConceptService#getConceptsByClass(String) may turn deadly in some cases? Because somehow the autocomplete mode hides the issue away?

Note that our real issue is not with the HFE tag that we would probably always use with autocomplete anyway ; it’s with the above Core method’s implementation.

When debugging why the new <condition/> tag was so slow, I realised that it’s because of fetching concepts by class, literally just this call:

conceptService.getConceptsByClass("Diagnosis")

And when the above one is hit many times (because of a <repeat/> for instance), then the form becomes unusable.


@samuel34 could you try in a test form to <repeat/> 15 times:

  1. this:
<obs conceptId="..." answerClasses="Diagnosis" style="autocomplete" />
  1. that
<obs conceptId="..." answerClasses="Diagnosis" />

And see whether you can reproduce the performance hit?

1 Like

Yeah @mksd, the ObsSubmissionElement is a beast to follow, but in the autocomplete use case that method is never hit, but instead it renders a DynamicAutocompleteWidget does a more targeted search.

The right thing to do, I think, would be to just fix the getConceptsByClass(ConceptClass) method… there’s really no reason to do a Lucene lookup here, right? There is no free-text searching we are doing here, correct? (it’s taking a ConceptClass, not a String). Unless I’m nissing something, this seems like a backportable fix…

Take care, Mark

Your right Mark!

True dat for the case of <obs/>

@mogoodrich absolutely, that’s my conclusion as well. This specific query should not be a Lucene query. I have recycled the ticket into this: