Backend support for draft forms/encounters

Thanks for your input @mseaton. What I’ve had in mind was actually something less invasive, maybe using some configurable attributes to make an entity support draft. But that 's just an idea.

@ibacher how is this, if at all, modeled in FHIR?

In many EHR note-writing systems or ordering systems, an item can be

  • Draft – not seen by anyone other than the author, not posted, not live
  • Preliminary – posted, but not ‘official’ – usually refers to something that requires a cosignature before it becomes part of the orders / part of the record
  • Final
1 Like

This is going to be a bit of a complicated answer, because FHIR’s equivalent of forms is modelled somewhat differently than in OpenMRS.

In FHIR, a form would be represented by a “Questionnaire” resource. The form-as-filled-in is represented as a “QuestionnaireResponse” resource. Each questionnaire response has a status which can indicate whether the response is “completed” or “in-progress”. A typical way to use this would be something like this:

  1. The user opens the questionnaire and begins filling it in.
  2. The user does something else, so the questionnaire response is saved in “in-progress” status.
  3. The user comes back to the questionnaire and finishes it. The questionnaire response is moved to “completed” status.
  4. Once the questionnaire response is moved to “completed” status, it is then handled by some backend process that converts the questionnaire response into the resulting observations, etc. (an example of this is the FHIR SDC’s $process-response operation).

The problem is that this requires some kind of intermediate state (a questionnaire response in the above outline; an HL7 message in the workflow Mike mentioned) persisted somewhere that the form engine is able to recognise and process.


For Orders, FHIR’s modelling is just an single attribute on the order, the “status” attribute that can be in “draft”, “active” or a few other statuses.

I think this is less invasive to the form entry engine, but not less invasive to the OpenMRS system overall, which will have lots of modules, functions, and features that have no knowledge of how to distinguish between a draft or completed encounter (and associated obs and orders). But if the form data is saved in pre-submitted, serialized state (eg. as a serialization of the form data, or the FHIR resources that will ultimately get submitted), then this can become a feature of the form entry engine (to support authoring draft encounters) in a way that doesn’t require changes to the rest of the application.

1 Like

This makes a lot of sense!

Adding draft encounters including all of its components (draft orders, draft encounter diagnoses, draft notes, etc.) would be wonderful and powerful, but is not a small change. As @mseaton points out, all the business on the backend and anything retrieving these data would need to be aware of the possibility of data being draft status. But I think that’s different from @mksrom’s requirement.

I believe @mksrom wants to be able to discriminate between signed (i.e., completed) forms and those that are not yet complete.

@mksrom, in Java we distinguish between saved & unsaved resources by determining if a UUID has been assigned. Am I correct in assuming that we’re running into this problem because the frontend is assigning UUIDs to resources before they’re written to the database (saved)?

Correct. And we would need that any user can come back later to the “not yet completed” form and continue filling it in. Or, finally decide to “Sign it”.

That would be just great! Though that’d mean the data saved in draft is nowhere shown in the application (as there is no encounter nor obs saved yet). But maybe that’s OK for our use case. I’ll check.

(btw @grace would it be possible to add this topic to next TAC?)

1 Like

Done :slight_smile: Is it okay to combine with the OpenEHR CDS topic?

Thanks. I would think that this draft encounters status is worth a 1h call, but let’s try to keep it short.

1 Like

Either CDS or draft encounters could easily fill the full hour. As it turns out the CDS discussion is scheduled for 25 April, so we can use next Monday to focus on draft forms/encounters.

1 Like

To clarify, there are two very different approaches being discussed here:

  1. Draft forms. An incomplete form is serialized and persisted (somewhere, location TBD) so it can be restored to its incomplete state and someone can finish it later. An encounter is not created until the form is finally submitted.

  2. Draft forms with draft encounters. An incomplete form is stored within an incomplete encounter. The encounter, any form(s) belonging to the encounter, and eventually any other encounter data (orders, diagnoses, etc.) would be stored alongside all other encounters, but marked as in “draft” status. This is the ‘just add a “draft” status to the existing encounter table’ @mseaton was referring to in his above comment.

We would love to have #2 (draft encounters) and will eventually need to tackle that, but, like @mseaton implied, it will take a significant amount of work to ensure the long tail of existing code is either made aware of the possibility of draft encounters or uses methods & SQL that filter out unfinished encounter data. But this is the “invasive” approach.

I believe what you are looking for is #1; however, we can’t store these incomplete “draft” forms within completed encounters (any encounter in the database is currently considered complete), so we’ll need to come up with another place to store them.

So, just got back with the client. Indeed that is not a problem.

Hello everyone, I know I’m late to the party but maybe have an interesting proposal.

I would propose persisting draft objects in an unserialized state. Any frontend module can dump draft blobs in JSON format to a /draft endpoint in some format like

#POST /draft
{
   module_name: 'form7',
   data: {JSON OBJECT}
}

being stored in table drafts with columns: [user_id (clinician, not patient), module_name, data]

and then retrieve data by calling GET /draft/{module_name}. When a clinician resumes working with a patient they can check the draft data and load up ui elements appropriately. When they save real data this goes through normal workflow as usual and /draft/{module_name} gets emptied with a DELETE

This has the advantage of 1. allowing the saving of unverified data if a clinician doesn’t have time to clean data to the point of database validation and 2. existing tables are left as-is, all data in those tables is always valid.

This technique could be extended to support encounters by adding an optional ‘encounter_id’ to the draft table and a query param in the GET like /draft/{module_name}?encounter={encounter_id} and a DELETE like /draft/encounter/{encounter_id}

Let me know if this is helpful

2 Likes

We had a good discussion on today’s TAC call. Some outcomes:

  • Draft forms or draft encounter?
    • While we’d like to have draft encounters (including support for draft orders, forms, encounter diagnosis, allergy changes, etc.), this will require significant changes to the API/Platform, since adding status to encounter that allows for “planned” encounters could easily break all existing code if it doesn’t take backwards-compatibility into account and we aren’t careful to identify queries in existing code that need refactoring.
  • What architecture to use?
    • We should consider the application state feature @zacbutko suggested as an architectural approach, since it could have multiple other uses.
    • For @mksrom’s use case, this would be something like /draft/patient/form (i.e., a draft form specific to a patient).
  • How long are drafts persisted?
    • Default behavior would be they are persisted forever. As long as each draft has a update_at timestamp, then local business rules could be applied to void or purge drafts.
  • Who can access drafts? Can they be shared?
    • Permissions can be applied on top of draft feature as long as we know who created the draft and for whom.
  • Making forms uneditable is a separate issue (@mksrom’s use case suggested forms would reach a state of being uneditable).
    • We would suggesting using permissions to control who can edit forms.
  • Design ideas
    • Form content
    • User (creator)
    • updated_at timestamp
    • Patient (optional)
    • We can’t really link to encounter, since we wouldn’t have encounter in common use case (not created until form validated/submitted/saved).

Here’s the recording:

5 Likes

Hi Everyone,

It’s been two months since we discussed this and it sounds like generally everyone is in favor of having draft encounters. It sounds like the best approach would be to implement it “for real” instead of just using a frontend cache.

What are the next steps here? How should we proceed to bring this feature to life?

Thanks, Zac

2 Likes

@zacbutko,

We discussed backend support for forms on yesterday’s Platform Team Meeting. I’m going to refer to this as “draft form data” (to distinguish from the ambiguity of “draft form”, which could refer to a definition of a form that hasn’t been completed).

  • We focus on support for draft form data
  • We need to agree on API endpoints
  • New API endpoints would be added via a Form Data module that would add REST endpoints by depending on & registering endpoints through the REST Web Services module (in a new FormDataService)
  • The basic properties of a draft form data would be
    • User
    • Patient
    • Which Form is associated with draft form data
    • Draft form data contents (serialized to JSON)
    • Timestamps for “created on” and “updated on”
  • We’re assuming draft form data will persist forever (i.e., until deleted) for now. Having timestamps will allow for TTLs to be applied later without having to be decided up front.

The next step is to agree on the endpoints needed. Our assumption would be:

  • CRUD (save, get, update, and delete draft form data)
    POST /formdata
    GET /formdata/:uuid
    PUT /formdata/:uuid ← our REST module might use POST here
    DELETE /formdata/:uuid
    
  • Search draft form data (filtered by patient, user, and/or form), for example:
    GET /formdata?patient=:uuid&user=:uuid
    GET /formdata?patient=:uuid&form=:uuid
    

Are there any other endpoints you would need?

@ibacher & @dkayiwa, please let me know if I misrepresented any of our conversation.


FormDataService

Initially, I thought these endpoints would belong in our FormService; however the OpenMRS FormService is focused on form definition management – i.e., metadata, not clinical data. After a little more thought, form data (like FHIR’s QuestionaireResponse ) represents data that are (or are going to be) part of an encounter, similar to observations, orders, encounter diagnoses, notes, etc. So, it might make more sense to introduce this as FormData a better fit for managing form data (even in draft) would be within the EncounterService or, better yet, within a new FormDataService to parallel ObsService, OrderService, etc.

/cc @dkayiwa @ibacher

2 Likes

Thanks for the recap @burke . It sounds like there is a solid plan in place. @dkayiwa , @ibacher , is there any capacity in the platform team to start work on this?

This is on my list of things to do either this week or next week, so yes…

One short-coming of our current forms model is that we don’t really persist the “form as answered” (which is what a QuestionaireResponse really is). This means that all our form engines end up doing work to go from data as persisted (Obs, Orders, etc.) back to data as entered. (Of course, the upside is that this means there’s no possibility of data errors where what was captured on the form and what became an Obs differ).

From our discussions, I think, at least for the immediate use, we’re going to end up storing something that looks more like the full encounter payload in the formdata service, since the AMPATH engine already knows how to translate an encounter payload into pre-entered data in a form.

1 Like

I’m digressing a little so feel free to ignore or branch out a new topic, but, would there be any value in at least recording just the actual question that was asked? As a middle ground between 1) what we’ve got now and 2) keeping a record of the entire questionnaire as it was used (which is what you alluded to).

Of course this would require to expand the Obs model to hold a member like “actual question asked” or “exact question asked”, whose value would always be in the actual language used on the form.

This not even a hard requirement from us. We haven’t go that far in the data science that someone asked what the actual question asked was for each recorded observation. But I remember @jdick once mentioning this as well over a call a while back.

1 Like

The original design for forms in OpenMRS was to record form & form_field definitions that would contain the actual questions asked, linking to the dictionary for concepts used and linking to obs for data generated.

We also envisioned adding “form data” as another type of data generated within an encounter (alongside obs, orders, diagnoses, and notes) to store arbitrary form data collected during an encounter. The idea was form data could (but would not have to) generate obs – you could generate obs for form data that you needed in flowsheets, decision support, or reports, and leave other form data (without generated obs) for auditing purposes (e.g., questions used for billing, to set attributes on the encounter/visit, or be used for workflows that didn’t need obs).

Our form service is focused on form design/definitions (like FHIR’s Questionnaire). A form data service could focus on storing/retrieving instances of data collected using a form (FHIR’s QuestionnaireResponse). We could link form data to obs generated from them (like we have currently done loosely with form_namespace_and_path so edits to form data could update associated obs when they exist.