REST Module and Multi-valued attributes

So I’ve been helping the ICRC troubleshoot an issue that’s been arising recently from the Angular form engine when a form with diagnoses is modified, however, the issues we’ve run into are only loosely related to that.

1. Handling Lists / Sets

First, we notice that whenever data in the form of a List or Set on the backend is submitted as an update, it is additive, meaning that the data in the request is always appended to the list. This means, for example, that if I submit an Encounter like this at create:

{
   // ...snip
   "encounterProviders": [{ "provider": "1234", "role": "Encounter Provider" }]
   // ...snip
}

And then, when updating the encounter, I submit the same thing:

{
   // ...snip
   "encounterProviders": [{ "provider": "1234", "role": "Encounter Provider" }]
   // ...snip
}

I end up with two EncounterProvider entries, assigning the same provider the same role in the same encounter.

This arises, in part, because BaseDelegatingResource#unchangedValue() doesn’t have any special handling for Lists or Sets, and particularly Lists or Sets of objects. This also means that there’s no way of working around the duplication by setting the UUID of encounterProviders entries if it already exists to avoid generating this duplication, because:

{
   // ...snip
   "encounterProviders": [{ "uuid": "8a413042-7024-44e7-8db6-da3e10496018" ,"provider": "1234", "role": "Encounter Provider" }]
   // ...snip
}

Will still be treated as a new entry, now with a duplicate UUID. It seems like the unchangedValue() method should have some handling for lists and sets of OpenmrsObjects, though this would likely be a breaking change at this point.

2. Flushing Lists / Sets

Since the List / Set is never treated as unchanged here, it means that we always end up calling BaseDelegatingResource#setProperty() with the value of the list. This, in turn, calls the #setPropertyWhichMayBeAHibernateCollection() function, which, if it gets a collection, will load the existing collection, call clear() and then addAll() to add the newly added items.

This results in Hibernate adding a new item to the persistent collection that needs to be flushed. However, because the main object isn’t flushed from the Hibernate session, it means that sub-properties of the main object may not be properly filled out. Concretely, for example, on creation, we accept encounterProviders as shown above:

{
   // ...snip
   "encounterProviders": [{ "provider": "1234", "role": "Encounter Provider" }]
   // ...snip
}

Even though this entry is missing required properties on the actual EncounterProvider object (specifically, a link to the containing encounter). In the normal course of things, this is handled by Hibernate’s automatic cascade when the encounter is saved, but if the Hibernate context is flushed before the encounter is saved, Hibernate ends up trying to create a new row in the encounter_providers table without an encounter_id field.

Unfortunately, this happens in the REST module because the encounter diagnoses are processed after the Encounter provider and can result in a call to getDiagnosisByUuid(), which starts a new Hibernate transaction and flushes the pending new Encounter Provider.

I’m not sure what the resolution here should be, so I’m raising this Talk post to discuss how we should resolve this.

CC: @ICRC

2 Likes

Do you mind explaining a bit, why this would be a breaking change instead of fix to a bug?

Could this be one of those cases where we have to set the hibernate flush mode to FlushMode.MANUAL?

Seems like that would solve it. I’m not quite sure where to apply that.

In the case I’m working on, appending extraneous encounterProviders is a clear bug and very annoying; however, because this method applies to everything that uses the REST module framework (or at least the DataDelegating part), it’s hard to guarantee that this is always a bug and since the last substantive change to the logic here was a decade ago, there could easily be someone relying on this functionality.

Would it help if we change the unchangedValue() method signature from private to protected such that sub classes can override the default behaviour where it does not make sense for them?

Would this still be an issue if Lists/Sets are correctly handled?

Yeah, I think that’s actually perfect.

Well, it would still be possible to add an item to an existing list and still call a getter. For example, if I submit something like:

{
   // ...snip
   "encounterProviders": [{ "provider": "1234", "role": "Encounter Provider" }, { "provider": "2345", "role": "Encounter Provider" }]
   // ...snip
}

Assuming the first of those already existed, but the second was new, it could still trigger the same condition, so I think it would remain a bug.

Hello Ian, thanks for this topic. For my understanding, is it related to the issue we face with some AOP Advices and when we try to persist the patients/encounters history for the current user ? Have a nice day

I don’t think it’s directly related to that. Everything I’ve seen so far is core OpenMRS.

Hello @ibacher and @dkayiwa it seems this resolution introduced a regression:

  • when we save a patient containing some attribute using an answerConceptSetUuid ( civil status, country of origin) as an answer, the value is saved with the concept 's uuid and no more the concept_id. If we try to reload the Patient details page, the value is empty ( the page should expects a concept_id and get a uuid)
  • no issue with OpenMRS V2 UI.

We did some tests by switching from the version 2.40.0 to 2.41.0 and every thing is ok with 2.40.0 and the issue is introduced with 2.41.0

Sorry it could come from this: omod-common/src/main/java/org/openmrs/module/webservices/rest/web/ConversionUtil.java Not related to these changes

Maybe this commit: RESTWS-901: Call property setters only once on map conversion (#581) · openmrs/openmrs-module-webservices.rest@cbcf07f · GitHub?

@frederic.deniger Are you able to provide an example of the REST call that’s doing this?

yes trying by reverting this commit.

@ibacher I confirm that by reverting the commit RESTWS-901 it works as expected.

can we propose a rollback ? Or do you know how to fix it completely ?

Hey @ibacher , Here is an example of the endpoint that is called, whose value is null. (Of course, the domain and the patient change with the environment) :

http://localhost:8080/openmrs/ws/rest/v1/person/36ee2f37-858c-4dab-bade-50479c72c299/attribute?v=custom:(uuid,display,attributeType:(uuid,display,format),value)

If I intercept the request by “injecting” the uuid concept into the response (value), it works as expected. Let me know if you need anything else.

Thanks,

CC: @frederic.deniger

Yeah, I think that’s the thing to do. I don’t really know how to undo the issue.

I tried to ping Ryan in the PR as well