Reporting module: only one visit is displayed from my evaluated DataSetDefinition

Hi guys!

(@mseaton)

I have few questions about the Reporting module to produce a simple visit report. (cf this other post)

First, I don’t understand the difference between Data and DataSet? I was under the impression that a DataSetDefinition is filled with data from the DataSetEvaluator class (thanks @mseaton) in which the actual retrieval of all the data is made.

(Q) So what is Data class used for?


For this post purpose, I will try to display the visitId and the patientAge of all visits for the month. In MyDataSetEvaluator I am iterating through the list of visit IDs previously retrieved from an HQL query which retruns 5 visit IDs:

Here is what I do next:

    public DataSet evaluate(DataSetDefinition dataSetDefinition, EvaluationContext context) throws EvaluationException {

        [...]      

        // Create the visit ID column
        DataSetColumn visitIdColumn = new DataSetColumn("visitId", "Visit ID", Integer.class);
        // Create the patient age column
        DataSetColumn patientAgeColumn = new DataSetColumn("patientAge", "Patient Age", Integer.class);

        for (int visitId : visitIds) {
            
            DataSetRow row = new DataSetRow();
            
            row.addColumnValue(visitIdColumn, visitId);
            row.addColumnValue(patientAgeColumn, visitService.getVisit(visitId).getPatient().getAge() + " year old");
            
            dataSet.addRow(row)
        }
        
        return dataSet;
        }

And this is the preview output of this dataset:

→ Only the first visit is displayed. → Rows and columns are switched.

Looks like I am missing something.

Thanks

Romain

@mksrom, that code snippet above does not show what kind of DataSet you are using, but it could be that your problem is that you are constructing and returning a MapDataSet, which would behave as you describe. Most data sets that you intend to return a table of data that is fully populated in the evaluator should use a SimpleDataSet. Hopefully this is the issue.

As for your question about the difference between DataSet and Data, the intention is that a data set is more a construct that would contain various data elements, whereas “Data” is intended to be a single data element. The genesis of the data/datadefinition classes was to solve a couple of use cases that repeatedly arose:

  • The need to represent re-usable data calculations that could be used in lots of different contexts (this came after the logic module, but before the calculation module, which developed as a way to further abstract this away from any dependence on the reporting module). For example, let’s say you wanted to be able to display the patient’s latest BMI on 4 different reports, on a patient summary screen, and in a clinical email alert. The idea was to be able to write the logic for this once in a PatientDataDefinition instance, and then to be able to evaluate and use that in those contexts repeatedly, and not duplicate code or risk having varying logic.

  • To support Row-Per-XYZ reports / data sets. Like you are doing above, the reporting module aims to support out-of-the-box row-per-domain-object reports across the OpenMRS spectrum. Row-per-person, row-per-patient, row-per-encounter, row-per-obs, row-per-visit, etc. These have been partially implemented based on the priorities and needs of those who have adopted it, but the pattern is well established. The idea here is that the rows are defined by implementations of the “Query” interface - PersonQuery, PatientQuery (ie. CohortDefinition), EncounterQuery, ObsQuery. These allow you to define which persons, patients, encounters, obs should be on the report. Then, the columns are defined by implementations of the “DataDefinition” interface - PersonData, PatientData, EncounterData, ObsData. And there are adapter classes for exposing, say, PatientData as EncounterData (since this is a 1-N relationship).

At it’s core, the goal is to support the building blocks that allow one to do lots of different types of reporting consistently, and to have the logic encapsulated in a single place for a given calculation. So, you can use a CohortDefinition (which is a PatientQuery) in an indicator report to get the count of patients that fit a certain criteria, and then use the came CohortDefinition in a row-per-patient report to export a custom set of data for each patient in that cohort. You can create a PatientDataDefinition that contains a complicated set of logic around if there are any critical alerts for that patient, and then you can use this definition to display patient-level alerts of the patient dashboard, to be output into a row-per-patient report for clinical care, etc.

Mike

Indeed. I replaced with SimpleDataSet and it works now. Thanks a lot :thumbsup:


In my dataset evaluator, I don’t use any existing data, but rather retrieve all required data right into the evaluator.

So it looks like it is not the way to go and that I should use existing Data. Correct? How could I achieve that? For instance, I will need to display Chief Complaint and Diagnoses. Is there any Data or DataSet that I could leverage to get those easily?

Does it mean that I don’t need to create my own dataset at all, but use an existing one instead?

In PIH Malawi module (or other module), do you know any example that is close to the result I am trying to get, ie, a One Row per Visit report that shows patient and observations data?

@mseaton could you be having any more pointers in regards to this? :slight_smile:

1 Like

@mksrom yes, the idea is that you should be able to use a built-in VisitDataSetDefinition (this is a row-per-visit dsd), and per the explanation above, you can control which visits you wish to appear via adding VisitQueries to the “rowFilters” property, and then which visit data you wish to display by adding VisitData, PatientData, or PersonData to the columnDefinitions property. See the “VisitDataSetDefinition” and it’s parent class “RowPerObjectDataSetDefinition”, and their corresponding evaluators.

Doing a quick usage search, I see that this has been utilized in the “coreapps” module to implement the “Awaiting Admission” page - you can see this here.

Give that a look over and let me know if this is something I can help you to explore and evaluate further.

Mike

@mksrom, also, for Chief Complaint and Diagnosis data, there are likely several possibilities depending on exactly what you want, but I’d look first at the ObsForPersonDataDefinition. Any Person Data Definition can be utilized within the ColumnDefinitions of a Visit Data Set Definition out-of-the-box, as there is a many-to-one relationship there. You can see the implementation of this (and how a PersonToVisitDataDefinition adapter class is used to achieve this) here.

The main current drawback you’ll find with using a VisitDataSetDefinition is that not many people have started leveraging it yet, so the universe of pre-built VisitDataDefinitions is rather small. It would be great to use this opportunity to ensure some additional basic VisitDataDefinitions were available, if you’re able to contribute those. I’m thinking things like “VisitDateDataDefinition”, “VisitLocationDataDefinition”, “ObsForVisitDataDefinition”, etc. There are similar definitions already written for Encounters that could largely be duplicated with very minor changes for Visits. These are very straight-forward to implement. In most cases we are talking about only a handful of lines of code.

Mike

What is the difference between a MapDataSet and a SimpleDataSet? When should I use one or the other?

Thanks for explaining all this. This are starting to organize already. However, some things are still unclear. Let me take the example of creating categories of age for each patient.

My report requires to show the patient’s age using 8 categories of age (8 columns) of which the definition is given. (This is a national MoH requirement and I can’t do anything about it)

Here is what I am doing (or trying to do :slight_smile:) :

  • Add a new AgeCategoriesDataDefinition that implements PersonDataDefinition
  • Create the AgeCategoriesDataEvaluator that retrieves the list of persons birthdate and creates the 8 categories for each patient based on the birthdate. This should return a map of 8 categories, each with their value (Yes or No)
  • Add a VisitDataSetDefinition to my ReportDefinition
  • Finally, because AgeCategoriesDataDefinition is implementing PersonDataDefinition, I should be able to assign it to my report’s VisitDataSetDefinition (thanks to the PersonToVisitDataDefinition that maps persons to visits as you have mentioned earlier)

Is the above the correct approach? :confused:

Thanks.

Romain

@mksrom, a MapDataSet is a specialized type of SimpleDataSet (it extends it), that can be used in circumstances where you know you will have exactly 1 row of data, and so it can provide additional convenience methods that make it easier to get that single row of data out. I believe is was mainly written for the use case of an Indicator Report, where you’d have a data set where the columns represented the Indicator, and there is a single row representing the value of the Indicator. SimpleDataSet should generally be the default choice.

Mike

1 Like

@mksrom, yes that sounds exactly right.

One area we haven’t covered yet is the concept of DataConverters. This is a construct we added that allows one to have a single PatientDataDefinition that can be converted in different ways for different column values. The idea is that the underlying calculation only needs to be done once (native caching ensures this), and then different columns can use this same calculation by converting it in different ways. When you add a columnDefinition to your VisitDataSetDefinition, you can specify one or more converters to apply to it.

In your case, I actually wrote some of the initial converters and data definitions with an age category use case in mind:

First, create an AgeDataDefinition (the evaluator for this will return an Age object for each patient)

Then, for each age range column in your dataset, add this AgeDataDefinition with a DataConverter that is able to convert from an Age to an appropriate value. There is an existing Converter that you could use to accomplish this out-of-the-box, which is the AgeRangeConverter. Or you can write your own converter following this pattern which is dead simple. You can see some examples of how these are used here and here.

Mike

1 Like

Oh how i wish there was a way of automatically converting this conversation into wiki documentation! :slight_smile:

OK thanks a lot @mseaton, looking into this right now.

While I am at it, there is something else I don’t understand :smiley: :

When adding a column to a dataset definition, the third argument required is String mappings.

vdsd.addColumn(name, dataDefinition, mappings, converters);

In your tests you sometimes set it to ‘(String) null’. In the actual code, it is often something really obscure :

ObjectUtil.toString(Mapped.straightThroughMappings(edd), “=”, “,”)

For instance:

dsd.addColumn(columnName, edd, ObjectUtil.toString(Mapped.straightThroughMappings(edd), "=", ","));

What is this “mappings”?

@mksrom - “mappings” is both a fundamentally important concept, and also one of the biggest complexities/confusions to fully understand the reporting module.

Mappings allow you to indicate how the parameter value of a parent definition should “map” to a parameter value of a child definition.

For example, let’s say you have a report and you want to be able to limit this report to patients in a certain age range.

First thing you would do is construct a new ReportDefinition that can take in the age range that you wish to limit the report to:

ReportDefinition rd = new ReportDefinition();
ageCd.addParameter(new Parameter("fromAge", "From Age", Integer.class));
ageCd.addParameter(new Parameter("toAge", "To Age", Integer.class));

Now, to limit this report, we can use an AgeCohortDefinition and set it as the “baseCohort” on the ReportDefinition. Perhaps we find that we had previously created such a definition, and it was defined as such:

AgeCohortDefinition ageCd = new AgeCohortDefinition();
ageCd.addParameter(new Parameter("minAge", "Min Age", Integer.class));
ageCd.addParameter(new Parameter("maxAge", "Max Age", Integer.class));

Now, we can’t simply add this CohortDefinition to the report. First off, it is not obvious how “fromAge” and “toAge” should relate to “minAge” and “maxAge”. To solve this, as you can see, the “baseCohort” property is not of type “CohortDefinition”. It is of type “Mapped”. This is because we need to be able to indicate how the “fromAge” and “toAge” parameter values we define on the ReportDefinition map to the “minAge” and “maxAge” parameters we have defined on the CohortDefinition.

We do this by creating a Mapped like such (in long form):

Mapped<CohortDefinition> mappedCd = new Mapped<CohortDefinition>();
mappedCd.setParameterizable(ageCd);
Map<String, Object> parameterMappings = new HashMap<String, Object>();
parameterMappings.put("minAge", "${fromAge}");
parameterMappings.put("maxAge", "${toAge}");
mappedCd.setParameterMappings(parameterMappings);

As you can see, we indicate that parameter values should pass through from parent to child definitions by mapping the parameter names.

There are various convenience methods around for reducing the amount of boilerplate code one needs to write for this, which includes what you refer to above (although this is even more obscure since it is converting to a Map<String, Object> which is then converted to a String representation of this, which will then be re-converted back to a Map in the method it is used in).

Commonly, the parent and child definitions have the same parameter names (eg. startDate, endDate, location, etc). When this is the case, the “straightThroughMappings” methods are a particularly useful convenience, including Mapped.mapStraightThrough(definition).

It should also be noted that mappings are not just limited to 1:1 parameter transfer. They also enable Mapping static values through. For example, in the above, if we always wanted to limit the report to patients between the age of 10-20, we could remove the Parameters from the ReportDefinition altogether, and change the parameterMappings to be:

parameterMappings.put("minAge", 10);
parameterMappings.put("maxAge", 20);

Mappings can also take in a limited range of expressions. The most typical use case for this is in date manipulation. For example, one could include the same DataSetDefinition that gets information about a particular date (eg. encounters on day), and include several of these in a single report. Something like:

rd.addDataSetDefinition("today", Mapped.map(encountersInPeriod, "startDate=${start_of_today},endDate=${end_of_today}"));
rd.addDataSetDefinition("yesterday", Mapped.map(encountersInPeriod, "startDate=${start_of_today-1d},endDate=${end_of_today-1d}"));

(start_of_today, and end_of_today are among a few pre-defined parameters that are always available. See here)

You can get a sense of what is possible by looking through this unit test.

Let me know if this is helpful. Mike

2 Likes

Thanks a lot for all these explanations. I am getting closer and closer to what I want. I did manage to add the age categories as you explained. (For anyone who might want to do this, here is the commit in our custom module https://github.com/mekomsolutions/openmrs-module-mksreports/commit/e38b9854ce2daf412af9523ee6813e7180b6cf44)

I have another question though :slight_smile: related to the mappings, and row filters.

So I am now trying to display only the visits that happened between a startDate and an endDate.

Following the example in CoreApps’s AwaitingAdmissionPageController.java and you reply above in the thread, I think I have to create a custom VisitQuery object that returns my list of visit IDs, in the form of VisitQueryResult, and add this VisitQuery object as a rowFilter of my DataSetDefinition.

So I create the VisitWithinDateRangeQuery class and its corresponding VisitWithinDateRangeQueryEvaluator.

VisitWithinDateRangeQuery

@Caching(strategy = ConfigurationPropertyCachingStrategy.class)
public class VisitWithinDateRangeQuery extends BaseQuery<Visit> implements VisitQuery {
  
  @ConfigurationProperty
  private Date startDate;
  
  @ConfigurationProperty
  private Date endDate;
  
  public Date getStartDate() {
    return startDate;
  }
  
  public void setStartDate(Date startDate) {
    this.startDate = startDate;
  }
  
  public Date getEndDate() {
    return endDate;
  }
  
  public void setEndDate(Date endDate) {
    this.endDate = endDate;
  } 
}

Is the @ConfigurationProperty annotation enough to define a class parameter as ‘mappable’?

VisitWithinDateRangeQueryEvaluator

@Handler(supports = VisitWithinDateRangeQuery.class)
public class VisitWithinDateRangeQueryEvaluator implements VisitQueryEvaluator {
  
  @Autowired
  EvaluationService evaluationService;
  
  @Override
  public Evaluated<VisitQuery> evaluate(VisitQuery visitQuery, EvaluationContext evaluationContext)
          throws EvaluationException {
    
    VisitWithinDateRangeQuery vq = (VisitWithinDateRangeQuery) visitQuery;
    
    evaluationContext.addParameterValue("startDate", vq.getStartDate());
    evaluationContext.addParameterValue("endDate", vq.getEndDate());
    
    List<Integer> visitIds = getAllVisitsWithinDateRange(evaluationService, evaluationContext);
    
    VisitQueryResult result = new VisitQueryResult(visitQuery, evaluationContext);
    result.add(visitIds.toArray(new Integer[visitIds.size()]));
    
    return result;
    
  }
  
  protected List<Integer> getAllVisitsWithinDateRange(EvaluationService evaluationService,
          EvaluationContext evaluationContext) {
    
        [...] Do the HQL stuff here that we don't care about [...]

    return visitIds;
    
  }
}

However when debugging this evaluator, the startDate and endDate parameters are always null. They are not passed correctly from my report params.

To pass them I am doing the following:

OutpatientRecordBook.java

@Component
public class OutpatientRecordBook extends BaseReportManager {

  [...]

  @Override
  public ReportDefinition constructReportDefinition() {

    [...]

    VisitDataSetDefinition vdsd = new VisitDataSetDefinition();
    rd.addDataSetDefinition("visits", Mapped.mapStraightThrough(vdsd));

    VisitWithinDateRangeQuery query = new VisitWithinDateRangeQuery();
    vdsd.addRowFilter(query, ObjectUtil.toString(Mapped.straightThroughMappings(query), "=", ","));

    [...]
  }
}

Something must be wrong in my mapping or configuration.

(I can’t take example on CoreApps on this because the mappings parameter is set to null:

dsd.addRowFilter(query, null);

The underlying evaluator can handle null parameters)

Thanks

Romain

@mksrom - the ConfigurationProperty annotation on startDate and endDate indicate that these properties are meaningful in determining the query results, and enable the framework to automatically handle things like Caching or providing UIs for editing all ConfigurationProperties on a definition, but they are not added as “Parameters” by default. The idea is that a Definition is capable of having it’s ConfigurationProperties configured directly (eg. by setting setStartDate(myDate)) or by having ConfigurationProperties added as Parameters to be specified at run time. It is this latter case that you need, and so you are missing the step of explicitly adding the ones you want to parameterize as parameters. It may seem a little redundant in this particular case, but you need to do this:

[…]

VisitDataSetDefinition vdsd = new VisitDataSetDefinition(); vdsd.addParameter(new Parameter(“startDate”, “Start Date”, Date.class)); vdsd.addParameter(new Parameter(“endDate”, “End Date”, Date.class)); rd.addDataSetDefinition(“visits”, Mapped.mapStraightThrough(vdsd));

VisitWithinDateRangeQuery query = new VisitWithinDateRangeQuery(); vdsd.addRowFilter(query, ObjectUtil.toString(Mapped.straightThroughMappings(query), “=”, “,”));

[…]

Incidentally, is the a particular reason why you are doing this in your evaluator?

It seems unnecessary. Why not just pass the VisitWithinDateRangeQuery object into that method directly?

Mike

Also, @mksrom, if you are building a generally re-usable VisitQuery that other users of the reporting module would value, please strongly consider adding this to the reporting module directly. In this case, I would almost certainly advocate just copying the BasicEncounterQuery / BasicEncounterQueryEvaluator classes into BasicVisitQuery / BasicVisitQueryEvaluator classes, removing the "forms’ property and replacing the “encounterTypes” property with a “visitTypes” property. The Evaluator is dead simple and would only require minor tweaking. See the use of the HqlQueryBuilder, which makes writing the necessary HQL with reporting module definitions super easy.

Mike

@mseaton, I have added the parameters to the VisitDataSetDefinition, but the startDate and endDate parameters are still null and therefore all visits are returned. I don’t understand why they don’t map.

OutpatientRecordBook.java#L93


About

I just did that for the unit tests at some point but yes I can just pass the VisitWithinDateRangeQuery object to the getAllVisitsWithinDateRange


Yes sure. I am happy to contribute to the Reporting module. So I’ll do that tomorrow and see if that solves my problem in the same time.

@mksrom - on line 93 that you linked to, you are correctly adding these parameters to the VisitDataSetDefinition, but what you are failing to do is to add the appropriate parameters to the VisitWithinDateRangeQuery.

So, line 96-97 would change to:

VisitWithinDateRangeQuery query = new VisitWithinDateRangeQuery(); query.setParameters(getParameters()); vdsd.addRowFilter(query, ObjectUtil.toString(Mapped.straightThroughMappings(query), “=”, “,”));

Mike

1 Like

Indeed, it works thanks! :thumbsup:

Hi @mseaton, I am still having little difficulties with parameter mappings… More precisely with the PersonAttributeDataDefinition that I use to display a ‘guardian’ attribute value of the patients (‘guardian’ person attribute type is provided as a report param). My dataset definition is a VisitDataSetDefinition.

@Component
public class OutpatientRecordBook extends BaseReportManager {

  ...

  private Parameter getGuardianNameParameter() {
    return new Parameter("guardian", "Guardian Name", PersonAttributeType.class);
  }

  ...

  @Override
  public List<Parameter> getParameters() {
    List<Parameter> params = new ArrayList<Parameter>();
    ...
    params.add(getGuardianNameParameter());
    ...
    return params;
  }

  @Override
  public ReportDefinition constructReportDefinition() {

    ...
    ReportDefinition rd = new ReportDefinition();
    rd.setParameters(getParameters());
    ...
    VisitDataSetDefinition vdsd = new VisitDataSetDefinition();
    vdsd.addParameters(getParameters());

    // Guardian Name
    PersonAttributeDataDefinition paDD = new PersonAttributeDataDefinition();
    
    Parameter personAttributeType = new Parameter("personAttributeType", "Person Attribute Type", PersonAttributeType.class);
    paDD.setParameters(Arrays.asList(personAttributeType));
    
    Map<String, Object> parameterMapping = new HashMap<String, Object>();
    parameterMapping.put("personAttributeType", "${guardian}");

    vdsd.addColumn(MessageUtil.translate("mksreports.report.outpatientRecordBook.guardianName.label"), paDD,
      ObjectUtil.toString(parameterMapping, "=", ","));

    ...

  }


}

(https://github.com/mekomsolutions/openmrs-module-mksreports/blob/unable_to_pass_parameters_in_paDD/api/src/main/java/org/openmrs/module/mksreports/reports/OutpatientRecordBook.java#L158-L168)

But when running the report, no value is returned, field is empty.

When debugging the PersonAttributeDataEvalutor here, the getPersonAttributeType() returns null though I correctly have selected it when running the report.

What am I not understanding? :slight_smile:

Thanks

Romain