Cohort Module Enhancements

[bholagabbar]

Hi @sharonvarghese, @maimoonak

I've been exploring the cohort module for quite some time now, have watched Sharon's complete video presentation on the module here on YouTube, deliberated over the additions and enhancements needed in the module and have come up with certain ideas of mine I would like to share.

Background:

I've been with OpenMRS for quite a while, fixing issues, attending Dev and Design Forums, helping out people on Talk and IRC and doing my best to contribute. Here is a comprehensive list of my code contributions and community interactions. Hope this qualifies me as a fairly good applicant

Discussion:

Right so I'm interested in working on the Cohort Module Enhancements Project. Our primary aim would be to get the cohort module as close to industry standard as possible. The work done by Sharon on this module as her past GSoC project is highly commendable. The improvements I would suggest are as follows:

1. Incorporate a Randomizer Component: Randomized Controlled Trials have by far proven to be one of the most effective ways to determine the success of studies conducted cohorts. A great article in this regard can be found here. This can prove to be an awesome feature to the module if implemented. I'd like to initiate a discussion on the same.

2. Integrate features of the Cohort Builder Module: The cohort builder allows us to query the DB for cohorts based on certain attributes. Using these features, It could become really easy to add cohorts instead of picking them up one by one. Details can be deliberated upon in discussion. (This falls under the spec '2. Extensive searched mechanism available over an extensive range of parameters' as mentioned by Maimoona in her description of the project page)

3. Complete the integration of the HTML Form Entry Module: This would probably be one of the top priorities of the project. A discussion in this context can be found here. @maimoonak You've mentioned that half of the work is already done. Could you elaborate what needs to be done to drive it to completion?

4. Redesigning the UI: Let's bootstrap it. Give it a modern '2.0' look. Improve the patient dashboard and take design inspiration from the Ref App.

5. Display Cohort Member Details: Modify the cohort dashboard to display the cohort members along with their details. We can first display the cohort and then implement a script that displays the individual cohort member details when hovered over this cohort. Just my initial ideas. However, this needs to be done either ways. No two ways there.

6. Integrate the Reporting Module to create reports for cohorts.

7. Finally, add this as an official module of OpenMRS after all the improvements and meeting all standards. (Official as in held by the OpenMRS github account)

There! Those ideas pretty much form the gist of my proposal I would prepare for the project. I'd like to discuss with Sharon and Maimoona what they feel should be given priority and how they'd spec out stuff.

Also, if possible, we could bring up the intended design of the Randomizer, HTML forms integration and incorporation of cohort and patient details on one of the design calls. I'd also like to hear inputs from @darius and @burke since you guys were involved in the design of this module too last last year.

I hope you found these ideas worthy and I'd really be up for discussing them in detail

Hoping for some great insights and brainstorming soon!

Reference Threads: 1, 2

EDIT: I was fortunate to catch @darius on IRC and got to discuss quite a few things in detail. Here is the chat log which covers the aspects of our discussion. @maimoonak, @sharonvarghese would be great if we could discuss what needs to be done further, assign priorities and start a fruitful discussion as soon as possible

[maimoonak]

@bholagabbar . I like the way you have done the preliminiary research and compiled the proposal. Below are our priorities:

1- Least priority.. Rightnow we donot have a usecase for this. Its would be a good addition if we have time.
2- Integration with CohortBuilder is good to have feature, but what I meant by the requirement is to have a UI where we can search data on the fly with different parameters like name, gender, cohort/household person belongs to, age range, differnt attributes, identifiers, some form type filled. If this can be achieved by integration in given amount of time it would be great. Usecase for this is team managing operations in field and wants to instantly respond a person`s query via phone by searchng the system.
3- Important and top priority.. The work done in last summer was to see the functionality and an addition to original requirements. We did not create proper and complete tags for cohorts and some features were not supported. You can compare the html form entry part of cohort module with original html form entry module.
4- 5- Important and need redesigning of UI and data organization improvements as well.
6- 2nd last priority
7- not in our priority list.

@sharonvarghese has done a good work in building powerful architecure so there would be minimal changes regarding that unless we add a new feature that needs DB redesign .

Considering the household data capturing where each household is cohort if you can provide mockups it would be very helpful..

[bholagabbar]

@maimoonak Thanks so much for the reply! I had been waiting for quite a while. Is there any place we can communicate frequently for eg through email where we can initiate a constant discussion?

Otherwise I often remain stalled on my ideas, owing to the fact that you happen to be a be a bit less active on talk Also, what should be my next steps and what would you be expecting in the proposal?

In regard to point 2 (Cohort Builder), I completely understood what you meant and the reason I suggested the cohort builder is because that is the closest we have to what we want (we are interested in the query part of the builder where as you mentioned, you can extensively search between parameters). We can take ideas from the Cohort Builder and implement them, or come up with a basic version ourselves. @darius Topic for one of the design forums maybe?

Regarding mockups, I believe you are expecting them for the UI redesign am I correct?

Hoping for a reply soon

[maimoonak]

Yeh, simple mockups showing data flow for different screens. Ideally it should have
- Starting screen where user would land just after login .
- Different tabs/navigable options for cohort management (metadata) VS day to day opertaional data management (Consider a Data Manager logging in and configuring metadata for Cohort / Programs available, Cohort Users/Roles etc VS Field/Hospital User who actually registers the patients see and can do)
- Forms which actually can fill the data (Should be HtmlFormEntry).

For next steps check out the timelines and schedules at https://summerofcode.withgoogle.com/how-it-works/

You might already have seen and read this https://wiki.openmrs.org/display/RES/Google+Summer+of+Code#GoogleSummerofCode-ExpectationsforStudents

For Proposal,
- Make sure the items are organized according to priority
- UI mockups can explain UI changes well..
- For backend changes if possible use any type of UML diagram you are comfortable with. If possible explore the code for cohort module, html form entry module, and cohort builder module and try making an estimate of changes required and then picture the time and effort an milestones. (I am more concerned about bacckend changes)
- If changes above are going to impact Data model mention it.
- Refer the existing cohort module features and what exactly we are going to change there.

I get email notifications for talk mentions and usually respond in time discussions but due to tight schedule it went down in the email list unread. Feel free to remind again if I donot respond within 24 hours.. . We can have discussion on skype later when work is started but for now to keep everything documented this forum is best.

[bholagabbar]

Thanks @maimoonak. I guess I should take about a few days for me to draft up the expected proposal you have in mind. Then we can deliberate of drafting the final ones. For the current one, I'll include the UI mockups, expected backend changes (I have NONE in mind so far..) and milestones.

I'll continue exploring the module, learning more about the HTML form entry and custom tags for cohorts and ofcourse, the cohort builder module.

Let's stay in touch

[bholagabbar]

Hi @maimoonak I have prepared my initial UI mockup for the cohort dashboard. The main change we wanted to accomplish was displaying details about the members in the cohort. I have tried and redone the UI keeping that in mind.

Also, I had a question regarding the tabs you wanted for Metadata vs Management. The thing is that a lot of the metadata options are interconnected within the already existing tabs(Dashboard, Metadata, Encounters (for eg, wouldn't add encounters here become adding metadata essentially?), Visits, Options). So do we really need to make this change?

• Patient Dashboard before search:

• Patient Dashboard after successful cohort search.

The second mockup diagram can act as the base for displaying Cohort Encounters, Cohort Visits, Cohort Observations as the pattern it is laid out will be the same.

The 2nd UI mockup I believe you would be looking for is the Extensive Search Mechanism feature for searching on the fly. I believe we can add this 'widget' of sorts on the Create Cohorts page. I am currently deliberating over it's layout and should come up with a mock UI for that soon..do tell me any more details you are expecting just in case.

Hope the UI was ok and we can go ahead with minimal changes if needed

@darius always looking forward to your advice regarding UI

[sharonvarghese]

Hi @bholagabbar

Happy to see your interest in Cohort Module Enhancements and @maimoonak has covered all the requirements in detail

[maimoonak]

@bholagabbar I must say that this is really good, thought through design. By cohort metadata I meant the data that is to support actual data entry. Like Cohort types, Roles, Programs, etc are metadata those are used in dropdowns to support data entry.

Moreover, I am also interested in seeing how we can display and edit data for patients for particular cohorts and where and how to provide adding new patients, edit existing patients and encounters would work!!?

Here are few basic requirements depicted by images attached to picture the minimum we want to see on new UI. This is just for specifying requirements and I guess we can trust your creativity .

[bholagabbar]

Thank you so much for the encouraging words @maimoonak

I have a few questions though.

1.
I've been looking at the module setup everyday and 2-3 of the pictures you have drawn are co-incidentally exactly the same as the existing UI.

Adding a new cohort -> Add Attributes -> Add members is exactly the order in which this appears in the module currently as well.

Ofcouse, I would bootstrap, change the UI, make it look coherent etc but how would I show it in the mockup? I think I am interpreting you wrongly here or something else. What I am saying is that we wouldn't be needing a mockup it the layout is going to remain just about the same..

2.

Also, I see that you have also added Integrate with Report module. Is that a high priority? I had HTML FE, Dashboard, Search Widget, UI in priority order, all to be done. Wonder where this fits in. While I'd love to do this and drive this project almost to completion, just wanted a heads up on as to what is expected for this

I guess I now have enough things to go on to make a proposal. I'll go ahead and make one, after clearing these doubts and then we can deliberate over the additions and improvements. I am currently creating UI mockups for the new pages where we are changing layouts. Do tell me if you would want the old one redone, how in context of my first point

[bholagabbar]

-what I meant by the requirement is to have a UI where we can search data
on the fly with different parameters like name, gender,
cohort/household person belongs to, age range, differnt attributes,
identifiers, some form type filled. If this can be achieved by
integration in given amount of time it would be great. Usecase for this
is team managing operations in field and wants to instantly respond a
person`s query via phone by searchng the system.

I have a doubt here: What data are we searching for exactly? If we want to see the patient and we know which cohort he/she is in, we simply go to view cohort and see patient details.

I believe you meant with several patient parameters, attributes etc, determine which cohort the patient is in ? Or you meant searching a patient with given parameters inside a selected cohort?

Note that in the second case the cohort is already specified and we are finding the patient and in the second case, vice versa.

[bholagabbar]

Refined UIs for the pages where major changes are required.

1.
Cohort Dashboard

2.
Manage Cohorts: This is the tab used by the implementer / admin. The people concerned with reading and using data stick to the first 2 tabs. Also, all the pages with plain multiple options can follow similar template. No separate UI mockup is needed IMO because it would simply become repetitive.
(Forgive me for the bad choice of icons )

3.
Edit Cohorts (on clicking Edit Existing Cohort): As you correctly mentioned, we needed a way to edit cohorts which I had completely ignored. Also, the current module has no way of deleting patients from a cohort. I hope you like this one. Did put in some effort here

[burke]

Great discussion here. I would urge you to consider creating a separate module for the enhanced feature or at least being careful to make a very clean separation between the API functionality and these additional features.

Part of our vision for cohorts, and the reason for generalizing the household module into a cohort module, was to work toward enhancing the cohort features of the platform:

• The ability to capture visits, encounters, and obs on cohorts just like we do now for patients – i.e., migrating these features from Cohort Module into core.
• The ability to define different types of cohorts (household, treatment group, research cohort, etc.) – migrating this feature from Cohort Module into core.
• StaticCohort and DynamicCohort as extensions of Cohort
  • Static cohorts are manually managed lists of patients (all we have in core now)
  • Dynamic cohorts are calculated lists of patients. To date, these have been implemented as part of reporting, but we'd like to make them a first-class feature of cohorts.
  • Clients that just need to consume cohorts (don't need to know or care how the list is generated) could inspect members via base Cohort class methods without needing to worry about static vs dynamic implementation details.
• For static cohorts, the ability to track membership over time (i.e., start & end dates on membership). This has been implemented in the Cohorts module.
• Cohort resources & operations accessible via REST API.

I think the design ideas in this thread look great, but I want everyone to be aware that there a a myriad of use cases for cohorts and its unlikely that all will be addressed through a single UI. In other words, we want to make sure that the API is solid and work toward getting those features into the platform. The cohort management UI(s) will continued to be supplied by modules like this one, but we'd expect there to be many different approaches & use cases. For example:

• Users could use a tool like the Cohort Builder to define dynamic and/or static cohorts.
• An outreach program might use a mobile application that creates cohorts for thousands of distinct households.
• A treatment program may use cohorts to record snapshots of who was in the treatment program at a given point in time
• A module could leverage cohorts to allow providers to define their own personal "patient groups" (e.g., "my diabetic patients" or "the list of patients I want to keep a close eye on over the next month")

These are just some random examples, but should help make it clear that a single, unified, cohort management UI will not meet everyone's needs. That said, there are definitely use cases for such management tools, so please don't let my comments slow you down; rather, just understand that yours are just the beginning of great ideas on how the community could leverage a robust cohorts API.

I've created an Expanded Cohort Features project to add support for static & dynamic cohorts and bring the Cohort module's date range for static cohorts into core as a first step toward achieving the vision mentioned above. It would be great to have a some real-time discussion (in a design forum) to coordinate these efforts.

[jdick]

To add on to Burke's post, we'd very much like to use the platform aspect of this module while building our own UI to support specific use cases for our organization. My only request for this module is that there are REST web services to support all interactions with the module.

[bholagabbar]

Be rest assured, REST web services are already an integral part of this module, even at present

[bholagabbar]

Thanks for the great insight @burke

I have read and analyzed your post and I believe that the major requirement you want to be taken care of is building a robust API to meet different needs and specs. Am I correct?

I will definitely discuss with @maimoonak the improvements and changes we can make on our side and in the proposal to meet the expectations you have put up.

I would be very willing to discuss this on a design forum on the 15th or on the 21st along with Maimoona, whatever date is convenient. I was just a little worried about the refactoring of Cohorts in Core since this is used in our module.

I definitely feel that we would have to assign and divide tasks related to cohorts between our two projects so we can do justice to both and we do this soon, so that we can do justice to proposals for both projects.

I believe that the cohort module provides great functionality and additions to cohorts in core and this is a great feature to have. Integration with HTML forms with custom tags (primary aim of project), building a cohort dashboard that can display cohort details in an effective way and of course cohort observations, encounters, visits, records etc are highly, highly useful features capable of catering to a large number of usecases.

[maimoonak]

1 - What I am saying is that we wouldn't be needing a mockup it the layout is going to remain just about the same..

Perfect then. The idea of sharing images was to make sure that UI based on different type of Cohort (Single Person or Multiple Person) is not missed somewhere in redesigning.

2- Also, I see that you have also added Integrate with Report module. Is that a high priority?

No, its not high priority if we can search data for existing cohorts. Also, I guess Burke`s module includes this feature.

Do tell me if you would want the old one redone, how in context of my first point

Bootstraping would automatically make UI better.

I guess I now have enough things to go on to make a proposal. I'll go ahead and make one, after clearing these doubts

Yes, please go ahead and the proposal can be changed until last date and also adding comments to document would be much easier

I have a doubt here: What data are we searching for exactly? If we want to see the patient and we know which cohort he/she is in, we simply go to view cohort and see patient details

Seaching could be on any parameter including cohort or program. We should be able to search by patient data or by cohort data as a cohort can have 1000 patients and a patient can be part of multiple cohorts..

As Burke suggested, we would keep the UI and API separate (as currently it is).
Also we would fing out if we are able to all type of serach and crud operations via REST on cohorts/members.

[bholagabbar]

Seaching could be on any parameter including cohort or program. We should be able to search by patient data or by cohort data as a cohort can have 1000 patients and a patient can be part of multiple cohorts..

So we can divide our search into Patient Search and Cohort Search correct? Can we keep this as the base? That means that the search is primarily in these 2 categories. We can design separate mockups for both

Also, did you like the new UI mockups?

[maimoonak]

Yeh. New mockups are good and looks promising.

Yes. Spliting it b/w two different part would make it easier and better managed.

Consider two different scenarios:
1- Search a person xyz with attribute D. Result should be all cohorts person belongs to and details of this person.

2- Search a cohort (household) with attrubute 123, address province abc, and or household head name matches Rid. The resultset should be households matching to given criteria and on clicking one household a details page should open that displays the HH details, members in HH and details of each member, and HH head info

[bholagabbar]

Thank you so much for all the guidance. I'm actually in the process of typing out a proposal. Do give me a couple of days.

I believe that we should coordinate our efforts with @burke on the 'Expanding Cohorts Module' over a design call on a suitable date. The main objective would be to diligently split the work between them

[sharonvarghese]

Thanks @burke and @jdick for your valuable inputs and its a great idea @burke for taking up expanded cohort @maimoonak and @bholagabbar great to see the discussion being taken forward and @bholagabbar I hope to see your proposal soon and as @maimoonak said we would be giving you our feedback on the same