I’m Isuranga Perera an undergraduate student at University of Colombo School of Computing. I participated GSoC 2017 and worked on implementing WS-Trust specification for WSO2 Identity Server where I had to work with Apache CXF and MSF4J(Micro-services Framework for Java). In addition to that I have worked on few features and Issues of Identity Server after the completion of GSoC 2017.
After going through the project objectives and guidelines I was able to come-up with some ideas for the project.
We can use a publisher subscriber architecture where users can subscribe to event while publishers can publish events. Specifically we can implement PubSubHubbub Specification. The idea behind PubSubHubbub is to push content rather than force clients to poll for it, which is typically how API implementations are designed to work. A setup includes these three elements:
Publisher - In our case this is the atom feed reader which reads for a certain event and publish(notify) it to the hub.
Hub - Subscribers can subscribe to a topic/event on hub which was created by a publisher(eg. patient returns to the medical institution event). As soon as the publisher notify about the event to the hub, it notifies that event to all subscribed users.
Subscriber - In our case this is the GUI and underlying logic where different users can subscribe to a particular event and get notified when that particular event has occurred.
AFAIK reading from the atom feed to find a particular event is not a good approach since it involves additional overhead. By using PubSubHubbub architecture at a later point in time we can publish events to the hub as soon as it occurs. As a example at the time of patient registration we simply can publish that event to the hub so that all users who have subscribed to the patient registration get notified. Here no addition overhead involves.
Appreciate if you can provide your feedback on the above ideas
In addition to that I have few queries to be made about the project
Currently is there any way to send notifications via email, sms etc. or is that also includes in the project scope
How do we recognize if a test result is normal or abnormal. Those information is provided with the test result or do we need to use some other approach
Hi @isuranga, thanks for your interest in the project, it’s great to see you’re giving some thought to how this might be implemented.
I think PubSubHub has evolved to become WebSub, if you plan to propose the use of this protocol it’d be good to use the most up to date spec. While the project suggests the use of the atom feed reader, there is scope to propose other approaches, as you’ve done here. However, it’s worth considering the advantages of the atom feed approach, and how it’s being used in Bahmni, before disregarding it. The atom feed also follows an event-based publisher/subscriber approach, but allows publishers and subscribers/consumers to be decoupled — the producer doesn’t need to know if the subscriber is up and running before it produces an event, and doesn’t need to know if the subscriber/consumer received the event that was produced. If you’re proposing an approach that delivers events to a subscriber, it’s important to address how it handles delivery failures and/or responses.
I can see some advantages to the underlying approach you’re suggesting - we want notifications to follow the occurrence/publishing of an event in as short a time as possible. With the current atom feed approach this could be achieved by reducing the polling frequency - if you feel the overhead of this is worth switching to a more synchronous approach, it’d be good to justify this in your proposal.
This is included in the scope of the project. In the long term, Bahmni would like to support multiple user-configurable notification types. But for this project, this can be limited to a single notification type, preferably one that is easy to implement and doesn’t impose any costs (e.g. probably not SMS).
The first priority is a notification for a ‘lab results are ready’ type event. But for extra credit, it could be useful to analyse the results of lab tests and build notifications around that. This primarily applies to lab tests with numeric values. It seems the OpenMRS wiki is down at the moment, but if you look here you should see details of the ‘Lab Test’ concept type. For numeric results, you can specify a normal value range (low/high) - abnormal test results would be those that are outside of this range. You can also read more about how to configure lab data in Bahmni.
CC’ing in @arjun and @vinay, they might have more to say on this.
Hi @danfuterman thanks for the clarifications and resources. I did some research on the points you’ve made here. AFAIK there are few downfalls in atom feed such as :
The client should always keep polling which involves higher resource consumption than in WebSub architecture
Atom entries are intended to be available indefinitely, the feed of events can grow very large over time. This makes the feed too large for consumers to conveniently GET over a network. Even though the feed can be separated into several feeds the problem still persists.
In WebSub architecture we don’t have to keep polling since it simply take usage of client side web-hooks to communicate with the client only when a new notification is available.
WebSub specification specifies a way to handle Hub to Subscriber delivery failures. Below paragraph extracted from the latest specification specifies it.
The subscriber’s callback URL MUST return an HTTP [RFC7231] 2xx response code to indicate a success. The subscriber’s callback URL MAY return an HTTP 410 code to indicate that the subscription has been deleted, and the hub MAY terminate the subscription if it receives that code as a response. The hub MUST consider all other subscriber response codes as failures; that means subscribers MUST NOT use HTTP redirects for moving subscriptions. Subscribers SHOULD respond to content distribution requests as quickly as possible; their success response code SHOULD only indicate receipt of the message, not acknowledgment that it was successfully processed by the subscriber. The response body from the subscriber MUST be ignored by the hub. Hubs SHOULD retry content distribution requests up to self-imposed limits on the number of times and the overall time period to retry. When the failing delivery exceeds the hub’s limits, the hub stops attempting to deliver that nofication. The hub MUST keep the subscription active until the end of the lease duration, and if a new update is published to the topic, MUST continue to retry delivery to the previously-failing subscriber.
In the specification they don’t specify a way to handle delivery failures from publisher to hub. In this case we’re allowed to define our own way. Here I propose using acknowledgements and retries. What we need to implement is a asynchronous publication service.
In proposing a new approach, it would be good to validate that these are real concerns with the current Bahmni implementation of atom feeds. This isn’t meant to discourage you from proposing new ways of doing things, but rather to validate that the problems you’re trying to solve are real and justify a shift in thinking. It’s also worth considering if these are downfalls or tradeoffs, and what has been done to address them (e.g. feed pagination).
It’s important to understand that the atom feed approach is already in use, and there are a number of existing feeds and tasks - when (certain) patient-related info is added to OpenMRS, this is published to a feed, and when a lab result is returned, this is added to a feed. If you want to see a list of the tasks for these feeds, log in to the OpenMRS admin backend on the demo site and view the scheduler tasks. These will not be done away with as they enable data sync between components in Bahmni. So your microservice could read events from these existing feeds, or build on top of this, preferably keeping redundancy and additional overhead to a minimum.
One last thing - this is a good resource to get a better understanding of the thought process behind adopting the atom feed approach for Bahmni.
@danfuterman Actually I missed this point . After going through the resource you provided now I realize the best way is to use Atom feed otherwise we’ll have to do major modifications to other components.
When it comes to project objectives
A second piece of the project involves creating a new microservice that reads the (existing) atom feed of patient events, and determines whether anything notifiable has happened. The default approach would be to build a Java application for this and use the atom feed reader library we have already built for use in Bahmni. But other tech approaches are possible. To start we would want you to implement one or two simple events (“patient is having a return visit”, “lab results ready”), but with a clear mechanism to allow for more kinds of notification to be added in future.
So here we basically have to do is design a atom client which keeps polling to the Atom feed and filter only required events. Please correct me if I’m wrong.
Hi @danfuterman I’ve come-up with few ideas for the proposal. Appreciate if you can provide a feedback.
Basically the module consist of 2 components.
Middle-ware
This provides several services to the user
Provides GUI which allow users to create new rules combining events and patients.
Group users depending based on events subscribed
Provide notification service support (email, sms etc.)
Communicate with FRM(Feed Reader Micro-service) upon the creation or deletion of events.
Communicate with FRM to retrieve new notifications
Feed Reader Micro-service
Micro-service reads the atom feed and filter events. We can separate this service from other modules so that this keeps reading the atom feed and when a new event is published it checks if that particular event is registered(users have registered for this event) with the service. If it is a registered event then the micro-service notifies about that event to the middle-ware otherwise it drops the event.To achieve this capability the micro-service should maintain a list of registered events. This is done by communicating with the middle-ware which keeps data about users, events that particular users have registered etc.
In addition to that Micro-service should be able to asynchronously communicate with the middle while polling for the ATOM feed.
Is there any preferred way to store subscription information ?. Do we have to persist them to the existing DB or do I have to come-up with some other solution (such as in memory data structure etc.)
Yes, for the most part this looks good, thanks! A few things to consider:
I’d avoid using the term middle-ware for this, and suggest going through the Bahmni docs to understand project structure and how/where the client and server-side code for those services would be added.
Remember that the notification mechanism for the project hasn’t been specified - this may or may not affect your high-level design, but the chosen mechanism should consider preferences (if any) of the Bahmni community.
Congratulations on being selected @isuranga, glad to be working with you on this project! Will be in touch soon to start discussing next steps and project milestones / timelines etc.
. After going through the resource you provided now I realize the best way is to use Atom feed otherwise we’ll have to do major modifications to other components.