radiology order lists, REST and pagination

Continuing the discussion from Standard way to implementing pagination in a module?:

Application Name: openmrs platform Version Number: currently 1.11.x but soon 2.x UI: legacy now and later with 2.x as well

Story/Question: Dear @raff and @pascal and other experts :slight_smile:

I have a question regarding lists/REST and pagination in modules.

Background: in radiology departments users of a RIS are working a lot with lists, lists of orders, reports, (with all different kinds of filters “today’s, last months’s studies”, …)… So lists are/will be an important part in the radiology module

I am not happy for how we handle lists of RadiologyOrder’s at the moment and am looking for guidance, so we get it right for other domain objects like for example RadiologyReport’s.

Current status: We use https://www.datatables.net/ for our lists. The radiologyOrderList.jsp shows a list of RadiologyOrder’s. This is done by including a javascript function which calls a controller that returns RadiologyOrder’s in an html table, so actually a view not json :frowning: . This table is then processed by datatables and the result put into another jsp.

Question: This procedure does not seem clean to me. Thoughts on solutions and concerns.

  1. REST: I am thinking of creating a REST service inside the radiology module which handles GET requests for RadiologyOrders including pagination. This REST service would then be called by the datatables. My concern here is authentication. Since we are still using the legacy UI and will for the foreseeable future how would I handle that? If I understand correctly REST services currently only work with basic http authentication? Or is this only meant for applications that only use OpenMRS REST and therefore have to take care of authentication? I am using the legacy UI, so is the call to the RestController already authenticated if done in the users active session from the legacy UI?

  2. Pagination: I understand that its not wise to just create a DAO like getAllRadiologyOrders() since users would soon get an out of memory error as amount of orders grows. How would I implement the pagination for the REST services? Can someone please point me to an example domain object of openmrs-core where I can see how this is done?

Thanks a lot!!

2 Likes

The REST module can do paging itself in memory (for resources with few items), or else rely on the service to do this (for resources with lots of data). See NeedsPaging and AlreadyPaged.

An example resource is drug. (this is a good example for the ‘search’ but not for ‘get all’). The openmrs-core search would look something like this for drugs, though a more standard method signature for openmrs data is here in Encounter.

1 Like

This sounds like a great idea to me! It will also enable developers to create an ecosystem of Radiology apps such as OWAs on top of your API. I think @judy might agree :slight_smile:.

Most resources in the Web Services Module use paging. For a simple example, see the doSearch and doGetAll methods of the Location resource.

I think if you have an active authenticated session, then calls to REST resources should work without requiring explicit authentication. Just like visiting REST URLs in your browser work if you are logged in. For example, visiting http://demo.openmrs.org/openmrs/ws/rest/v1/location will not require authentication if you are logged in at demo.openmrs.org.

Further, you can confirm this by opening up the console on pretty much any page on demo.openmrs.org and running:

jQuery.get('http://demo.openmrs.org/openmrs/ws/rest/v1/location', function(data) {
    console.log(data);
});

This should work without you explicitly having to provide login details.

1 Like

thank you both for your explanations!

I thought I repeat the question I asked you on irc here, since I am having some doubts.

My question was: I have a RadiologyOrderSearchHandler which gives me all RadiologyOrder’s for a given patient. I am using NeedsPaging to return the results. Now I consume the REST service using datatables with ajax in the legacyui, works well! The only challenge I have left is pagination. datatables expect the json response to contain “recordsTotal”, see https://datatables.net/manual/server-side do you have a suggestion for me on how to achieve that?

Your answer was:

I don’t think you can get the total number of resources from the API at the moment. you may need to do this: http://stackoverflow.com/a/33638608/1723879 alternatively, create a resource that returns the count.

By creating a separate resource you mean I create for example a RadiologyOrderListResource which has as property results containing the RadiologyOrder’s and recordsTotal? how is recordsTotal being populated? I assume I need to either provide a RadiologyOrderService.getRadiologyOrdersByPatientCount(Patient) or execute a .size() on the RadiologyOrderService.getRadiologyOrdersByPatient(Patient) result. Where does this go?

Is this “magic” of translating Resources into the actual JSON result done by a Converter? Would I then need to write an own Converter?

Maybe as a little background for the moment my RadiologyOrderSearchHandler only has one SearchQuery withRequiredParameters("patient"). But as a next step I want to expand on that, adding different kinds of SearchQueries/Paramters so users can for example get all RadiologyOrder’s that where ordered within a specific date range. Not sure if that would make any difference.

@pascal I think understood a little better right after I posted :wink:

So to your idea of another resource: I create another resource which I call RadiologyOrderCountResource which has its own RadiologyOrderCountSearchHandler which provides the same type of queries as does the RadiologyOrderSearchHandler but only returns results with property recordsTotal. So I will have to make two REST calls with the same parameters, one for the count, second for the data, right?

would you suggest any specific naming convention? if the actual resource is radiologyorder then radiologyordercount

Another idea I had was that maybe I can use the links property in the response and if there is a next show a next button and if there is a previous show a previous button and leave the total count for now.

Furthermore I wanted to ask if there is a need for a count provided by OpenMRS REST services? I assume my use case is a generic one and others would probably benefit from such a property. Should we think of a way to add such a feature to the REST module?

I don’t think there is currently any convention for getting counts. There is an idea here about using a custom HTTP header, but it might also be nice to use a subresource like GET /radiologyorder/count.

We should ask @darius, @wyclif, @raff & @dkayiwa for their opinions, or even discuss it on a design call.

@teleivo, are you developing in AngularJS? If yes, then have a look at http://rkorytkowski.github.io/openmrs-owa-conceptdictionary/#/class It’s a component similar to datatables, which you can find in https://github.com/PawelGutkowski/openmrs-contrib-uicommons#list-component It can handle any openmrs resource. You can see a dev demo at https://www.youtube.com/watch?v=Nii9x0Dq-6I&feature=youtu.be (soon to be published under OpenMRS channel) It can be used not only for OWAs.

Back to your original question. Yes, the count feature would be a good addition to the rest object returned for multiple results. Since we decided to return “results” and “links” inside response body, we could add “totalCount” to the returned object as well. Most of our java api methods do not expose count so it would be more resource consuming to get the full count of results for some resources (count in memory). In addition in most cases it requires another db query to get the count. I think we should return it when specifically asked by REST client, which for example would append totalCount=true to request parameters.

On Pascal’s comment about adding a count resource or subresource I think the benefit of embedding totalCount in a response itself is that we can return it for virtually any query, e.g. implemented using SearchHandler.

1 Like

Unfortunately no, no angular. I am developing on legacyui but thanks for the hint!

To me adding a new property to the request itself seems cleaner. But as you pointed out its probably best to only return it if explicitly requested via a parameter.

1 Like

Would you be able to work on that and/or create an issue in RESTWS describing the approach?

1 Like

I currently don’t have time. I might be able to provide someone who could implement it, an intern starting in July but cant say until 1 week from now.

Can you give me a starting point as to where this property would need to be added? I can then write up an issue in RESTWS.

1 Like

Start from here https://github.com/openmrs/openmrs-module-webservices.rest/blob/69de687b1e765e9c63aa79d37c5d3a5045f5bc71/omod-common/src/main/java/org/openmrs/module/webservices/rest/web/resource/impl/BasePageableResult.java

2 Likes

Dear @raff

just brainstorming: would following make sense:

*add method getTotalCount() in BasePageableResult (could return null so we don’t break compatibility and will be overridden by subclasses who do support this)

  • in BasePageableResult.toSimpleObject() add property totalCount to ret if Context contains a request parameter totalCount=true
  • in NeedsPaging override getTotalCount returning unpagedResult.size()
  • in AlreadyPaged add member totalCount with a new constructor where I can initialize the member.

And then it is up to the SearchHanlder to populate the totalCount

Just want to know if I am thinking in a good direction. Thanks!!

1 Like

Adding count to the response data sounds fine to me

1 Like

@teleivo, it’s perfect! Not much left to implement :slight_smile:

1 Like

dear @raff, I just opened the ticket https://issues.openmrs.org/browse/RESTWS-601 Please check if any info is missing.

I might give it a shot myself, since @ivange94 and I could really use this feature :slight_smile:

thanks for your help!

2 Likes

@raff I have implemented what I described in the issue see at https://github.com/teleivo/openmrs-module-webservices.rest/tree/RESTWS-601

I added a test for NeedsPaging, will add another for AlreadyPaged tomorrow and test the feature against the radiology module.

Can you please have a look and maybe comment also on testing, so I can incorporate your feedback. Thanks a lot!

1 Like

@raff just updated the PR, I tested the new feature with the radiology module (RadiologyOrderSearchHandler uses NeedsPaging to return results, the totalCount is returned if requested), see image

datatables happy, @teleivo happy :slight_smile:

Well done! Merged in!

nice, thank you all!