Cohorts and Cohort Searches

@reddberckley I can talk about this tomorrow (Thursday), so let’s set up a call for then instead of waiting for a design call. (I am in India time zone, so sometime in the morning Africa-time.)

The quick answer is that:

  1. the available queries are based on the java classes in this folder: https://github.com/openmrs/openmrs-module-reporting/tree/master/api/src/main/java/org/openmrs/module/reporting/cohort/definition. The field names would match the field names in the java class.
  2. The way that I produced the xml myself was by writing a throwaway junit test in the reporting module where I constructed the query in java code and asked the reporting module to serialize it, something like what is done here: https://github.com/openmrs/openmrs-module-reporting/blob/master/api-tests/src/test/java/org/openmrs/module/reporting/serializer/ReportingSerializerTest.java#L94
  3. We need to introduce a better way to do this. I would like you to please imagine what a better endpoint would be, and suggest it.

@darius Thank you. We’d like about 3:30pm your time. being 11:00am here and 1:00pm in Uganda, so we can have @dkayiwa and @ningosi join in. Does this time work for everyone?

That time works for me.

-Darius

@darius @dkayiwa @mseaton @ningosi or anyone who is interested. Here’s the link to the call. https://hangouts.google.com/hangouts/_/andela.com/design-call .

Here are the code snippets from today’s call:

To see a list of all the available cohort definitions in all cohort definition libraries: https://demo.openmrs.org/openmrs/ws/rest/v1/reportingrest/definitionlibrary?q=cohort

To produce what the serializedXml should look like, by configuring a Cohort Definition in Java (but due to our later conversation you don’t need to do this):

The existing REST API is

POST .../cohort
{
	serializedXml: "..."
}

The improved API I will work on this weekend will be:

POST .../cohort
{
  definitionLibrary: {
  	libraryKey: "reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate",
  	parameterValues: {
		"minAge": 15
    }
  }  
}

POST .../cohort
{
  composition: {
  	compositionString: "1 AND 2",
  	searches: {
  	  "1": {
        libraryKey: "reporting.library.cohortDefinition.builtIn.males"
      },
      "2": {
        libraryKey: "reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate",
        parameters: {
          "minAge": 15
        }
      }
  	}
  }
}

@reddberckley, @femi, and all,

Yesterday I went through and documented all the existing behavior in the Reporting REST module. That documentation is now here: https://github.com/openmrs/openmrs-module-reportingrest/wiki. Please bookmark this and continue to let me know of gaps in the functionality.

Further, I discovered that I had already built an endpoint that has equivalent functionality to the new API features I was going to add, although the syntax might be a bit trickier. Take a look at the adhocquery resource in the documentation.


Instead of this:

POST .../cohort
{
  definitionLibrary: {
  	libraryKey: "reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate",
  	parameterValues: {
		"minAge": 15
    }
  }  
}

…you can do

GET .../cohort/reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate?minAge=15

…and in the snapshot version of the module you can also do:

POST .../cohort/reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate
{
    "minAge": 15
}

Instead of this

POST .../cohort
{
  composition: {
  	compositionString: "1 AND 2",
  	searches: {
  	  "1": {
        libraryKey: "reporting.library.cohortDefinition.builtIn.males"
      },
      "2": {
        libraryKey: "reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate",
        parameters: {
          "minAge": 15
        }
      }
  	}
  }
}

You can do this:

POST .../adhocquery?v=rowFilters
{
    "type": "org.openmrs.module.reporting.dataset.definition.PatientDataSetDefinition",
    "rowFilters": [
        {
            "type": "org.openmrs.module.reporting.cohort.definition.CohortDefinition",
            "key": "reporting.library.cohortDefinition.builtIn.males"
        },
        {
            "type": "org.openmrs.module.reporting.cohort.definition.CohortDefinition",
            "key": "reporting.library.cohortDefinition.builtIn.atLeastAgeOnDate",
            "parameterValues": {
                "minAge": 15
            }
        }
    ],
    "customRowFilterCombination": "1 AND 2" // AND of everything together is the default behavior
}

This syntax is a bit worse, so maybe I will go ahead and implement the shorthand that we talked about anyway. If I do that I’ll post again. (Edit: I created REPORT-796 for this.)

But for now this should be sufficient for doing any of the queries you need in the cohort builder, and it should work in the current module version. Plus, if you add a couple columns and use the ?v=preview flag this will let you build a nice UI that shows details about the first 10 results, without adding a ton of server load.

1 Like

Thanks @darius we’ll work with this and give you feedback. Thanks a lot

Hi @darius we tried using the /adhocquery?v=rowFilters

{
  "type": "org.openmrs.module.reporting.dataset.definition.PatientDataSetDefinition",
  "customRowFilterCombination": "1 AND 2",
  "rowFilters": [
    {
      "type": "org.openmrs.module.reporting.cohort.definition.CohortDefinition",
      "key": "reporting.library.cohortDefinition.builtIn.females"
    },
    {
      "type": "org.openmrs.module.reporting.cohort.definition.CohortDefinition",
      "key": "reporting.library.cohortDefinition.builtIn.ageRangeOnDate",
      "parameterValues": {
        "minAge": 50,
        "maxAge": 60,
        "effectiveDate": "2017-03-27"
      }
    }
  ]
}

But when we used it, we got this error https://pastebin.com/XJHFt4T0. We have also used the atLeastAgeOnDate but we got the same error. We’ve tried it with the 1.7-SNAPSHOT and 1.8.0-SNAPSHOT version of the reportingrest module.

Very strange. If you look deeply in the error you’ll see

Caused by: java.lang.IllegalArgumentException: Failed to find item=50 using resource=org.openmrs.module.webservices.rest.web.v1_0.resource.openmrs1_10.OrderEntryConfigResource1_10@543e9e25

and this doesn’t seem like it should have anything to do with the query you’re running.

What server are you executing this on? (I’d like to try replicating it if it’s on a public server.)

tried it on qa-refapp.openmrs.org/openmrs and locally.

I committed a fix for this issue in the webservices.rest module. Ticket is here: https://issues.openmrs.org/browse/RESTWS-652

As soon as this makes its way through the CI pipeline, things should work on the qa-refapp server.

thanks. We will follow up on this.

@reddberckley, let me know how things turned out.

Hi @darius. We’ve been able to make searches for Patient attributes and we anticipate being able to do so for most of the other components. We anticipate having an issue with Drug Order as you pointed out earlier. We don’t think the existing cohort definitions would suffice for that. We want to focus on completing the others at the moment but we would appreciate hearing your thoughts on that and any other tab that may be tricky to implement.

We’ve uploaded the Patient attribute search at http://qa-refapp.openmrs.org/openmrs/owa/cohortbuilder/index.html#/

Cool! I love to see search results in OpenMRS. :slight_smile:

I happened to notice a small bug, which I’ll mention (though I assume this would be covered by some other story that is already in your backlog):

  1. search for age between 0 and 50
  2. click “Next” 5 times
  3. now search for age between 0 and 1
  4. You won’t see any results, and the paging display will say “6 of 4”

In general, I hope that when someone submits a search: (a) it adds a row to the Search History table, (b) it clears the search form that you submitted.

1 Like

Hello @darius, thank you for spotting this. We’ll have to create a new issue to fix that interesting bug. :smile:
Also, we’ll look into how to populate the search history table and clearing of the form fields. However, we’re currently looking at performing as many searches as possible.

Thank you once again.

Hi @darius the link for the wiki (https://github.com/openmrs/openmrs-module-reportingrest/wiki) is no longer accessible. It just redirects to the repo. I don’t know what’s wrong. Thanks

Someone had turned the wiki off in the project settings. I don’t know who or why.

I have turned it back on, and the documentation is available again.

-Darius

Hi @darius sequel to the demo call we had, I’ve made a list of all the searches we’ve been able to do and those that we’ve not been able to do

Searches that we have done Patient Attributes

  • Search patients by gender
  • Search by age (both minimum & maximum age)
  • Search patients by using date of birth
  • Search patients that are dead / alive or both
  • Search using a combination of 2 or more of the above fields
  • Search by using attributes such as citizenship, birth place, civil status, Mother’s name, race etc

Encounter search

  • Search patients based on encounter
  • Search patients with encounter at a particular location
  • Search patients with a minimum number of encounter
  • Search with a maximum number of encounter

Searches that we can’t do

  1. Concept & Observation a. We can’t see a cohort definition that supports searching patients who have (or does not have) a particular observation

  2. Encounter a. There is no parameter that supports fields like (within the last months and days, since and until). We had a workaround as regards this by using date but we don’t know if that will suffice. c. Search by form also generates an error, the link to the error is here https://pastebin.com/xPnM9Qs7. The encounterAdvancedSearch definition library key was added here https://github.com/openmrs/openmrs-module-reporting/pull/140/files

  3. Programme Enrollment a. No definition library key

  4. Drug Order a. No cohort definition

cc @mseaton

Hi @femi,

There are more options available for these needs:

These certainly exist. You should be able to use NumericObsCohortDefinition, CodedObsCohortDefinition, DateObsCohortDefinition, and TextObsCohortDefinition

This is possible by use of expressions when mapping parameters. There is some explanation of what is available to do this here. It is also possible by creating an appropriate definition library that takes in the number of months/days and units, and converts this to a date.

Based on the error message I see here:

java.lang.NullPointerException\n\tat org.openmrs.module.reporting.report.util.SqlUtils.openmrsObjectIdListHelper(SqlUtils.java:213)\n\tat org.openmrs.module.reporting.cohort.query.db.hibernate.HibernateCohortQueryDAO.getPatientsHavingEncounters(HibernateCohortQueryDAO.java:567)\n\tat org.openmrs.module.reporting.cohort.query.service.CohortQueryServiceImpl.getPatientsHavingEncounters(CohortQueryServiceImpl.java:156)\n\tat

My best guess is that this is caused more by bad test data than by any problem in the reporting framework. Were you testing this on patients and encounters that were previously saved, or just created in memory and not yet persisted?

I’m not sure what this means, but we should be able to just add in more library methods to meet our needs. There are several program-related cohort definitions available: ProgramEnrollmentCohortDefinition, PatientStateCohortDefinition, InProgramCohortDefinition, InStateCohortDefinition.

This does appear to be missing. We’d need to ticket this and get it added to the reporting framework.

Hope this helps a little. I’m on vacation this week, so I may be slower following up than normal. @darius FYI.

Mike

1 Like