Optimizing the Backend: Examples of slow responses in O3

Development
, ,
36

So, this is just going to be an unedited dump of some thoughts on performance. Apologies.

  • The Visits view in the O3 frontend as it currently exists is unworkable.

    To power the Visits view (which is very cool), we currently use a call like the following:

    /ws/rest/v1/visit?patient=<patientUuid>&v=custom:(uuid,encounters:(uuid,diagnoses:(uuid,display,rank,diagnosis),form:(uuid,display),encounterDatetime,orders:full,obs:full,encounterType:(uuid,display,viewPrivilege,editPrivilege),encounterProviders:(uuid,display,encounterRole:(uuid,display),provider:(uuid,person:(uuid,display)))),visitType:(uuid,name,display),startDatetime,stopDatetime,patient,attributes:(attributeType:ref,display,uuid,value)&limit=5

    For one of the demo patients (Betty Williams - 7521943e-dd1a-4e27-9b29-bb4241c52bef) each page of 5 results takes 20 seconds to return to my computer (which, it’s well-established that dev3 → me is faster than it is for most people). I think the issue here is that the query requires a large number of joins, which, in turn, results in a large number of individual queries being run on the backend. Hopefully, we could create an API endpoint customised to getting just the subset of data we need for this that would perform better than that (we’ve had reports of this taking over 60 seconds with real-world data, i.e., this vaguely-worded ticket). The data returned is also on the order of 2.5MB, which seems excessive. (This query also doesn’t properly implement pagination, so it just keeps requesting the same 5 visits again and again and again).

  • Not directly performance, but the call here to get the list of concepts for the test page doesn’t seem to be working correctly. Specifically, it’s not actually chunking the concepts into sets of 10 as intended, but just sending 4 batches of requests for 40 different concepts, all of which fail with an HTTP 414 error (URI Too Large)

  • Marginally performance related: it would be nice if the IDGen identifiersource endpoint could take multiple identifiertype arguments and return all the sources that match any of them. Currently, we need to do one request per identifier type, which is not ideal, especially as the overhead of individual requests often contributes to the perceived slowness of the app.

  • We end up with a lot of calls like this: /ws/fhir2/R4/Observation?subject:Patient=<patientUuid>&code=5085AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5086AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5087AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5088AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5092AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5090AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5089AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C5242AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C165095AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA%2C1343AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA&_summary=data&_sort=-date&_count=100. It think this is from the vitals and biometrics app. Each of those requests takes on the order of 2 seconds, mostly processing server-side.

  • We have requests like this: /ws/fhir2/R4/Condition?patient=<patientUuid>&_count=100&_summary=data that are not nearly as bad, but still take between 600ms and 700ms to return.

  • Right now, the query to get the list of concepts associated with Vital Signs looks like this: /ws/rest/v1/concept/?q=VITALS%20SIGNS&v=custom:(setMembers:(uuid,display,hiNormal,hiAbsolute,hiCritical,lowNormal,lowAbsolute,lowCritical,units)). This works, and it’s not per se a performance issue, but it only works on the assumption that there is just one concept in the dictionary named “Vital Signs” and that its a concept set. It would be nice if we had an API to request a concept by exact name.

  • We definitely need some kind of guidance about setting up caching properly on the server, because the O3 frontend really depends on files being cached for a very long time to be performant.

Since bandwidth usage has also been brought up: my run through with no caching on the first request and caching for everything else resulted in 7.61 MB of data over the wire of which 6.19 MB was code, CSS and fonts, meaning that the actual use of the app accounted for 1.52 MB or ~25% of the total transferred. (If you’re wondering how that squares with my assertion that the visit endpoint alone accounts for 2.5MB per request, well, it was both cached and all requests end up GZipp’ed so it’s 2.5MB of data, but only 225kB of data over the wire).

cc: @janflowers @paul

4 Likes