Preservation of Complex Obs data

Tags: #<Tag:0x00007fe2e68508d8> #<Tag:0x00007fe2e6850798>

I believe the consensus in healthcare is that medical data should always be preserved with records of corrections (except via some government established procedure if there is ever a valid requirement to purge data). Generally, I believe the policy is to preserve data, not to “clean up hard drive space”.

Currently, some types of observations like attached images (technically considered “Complex Obs”) can be deleted from the file system when the database Observation is voided. I have done some research on JIRA and see this specific issue has gone through some changes over the years. There appears to have been some fluctuation within the community between one approach (saving the files) and the other (deleting them). I’d like to hear what the community needs and policy are, to meet the needs of users and patients, and not just what an individual dev or group believes is “right” from their point of view. I’d appreciate any feedback, existing policies or case studies to direct future work in this area. Being that I’m primarily acquainted with the dev community, I’d like to invite comments from @AdvisoryCouncil , @janflowers , @dkayiwa , @wyclif , @mksd , and @mogoodrich .

I have described the current issue I am seeing here on a previous bug case that appears to have the same symptom for anyone technically inclined to read more: https://issues.openmrs.org/browse/LUI-135?focusedCommentId=259299&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-259299

Thanks all, Ken Long @long27km

1 Like

@lober @akanter @paul @burke Thought you all might be interested in thinking through this question.

1 Like

I think the underlying issue is regulatory requirements, which are complex, rather than preferences. Always good to look at regulations and decide if they have value, rather than simply rejecting them because they are from a different jurisdiction and compliance is not mandated.

Here’s a fairly random link which conveys the complexity of records retention in the US - different periods based on record type, federal guidelines, state law, and age of patient, to name a few.

While the EMR is not the only element of a medical record, increasingly it contains information which is not found elsewhere (ie., only an electronic copy exists). So, depending on the setting, it should not really be an issue of a preference to preserve changes or clean up the database - it should be an issue of regulatory compliance.

Beyond regulation, I think it is also sound practice to be able to reconstruct the record, including deleted items, whether through an ever-growing database of data which includes data “marked as deleted”, or whether through an ever-growing and archived audit log.

Thanks for the thinking around this, Ken.

Generally, our early designs focused upon never “deleting” data per se… instead, voiding data simply removes it from the user experience, but doesn’t physically delete it from the database.

My instinct is that if complex obs are deleted with a void, then that is an oversight that should be remedied.

@burke: thoughts? :slight_smile:

I also believe that we should preserve this data. And it looks like the direction code comments like this are driving towards: https://github.com/openmrs/openmrs-core/blob/master/api/src/main/java/org/openmrs/api/ObsService.java#L169-L171

I see that there’s some discussion on the JIRA ticket that we should create a global property that allows for configuration to either keep or delete voided complex obs files. I think that OpenMRS specifically here should take the stance with our products of “do no harm”, and not necessarily allow the deletion of data with our products used directly from our codebase. They certainly could modify their code to do so, but would like to make it more of a hurdle for someone to do that than just a configuration difference. @burke @paul @dkayiwa how do you feel about that?

I have no objection to that.

2 Likes