Discussion: Extending the Data Filter Module for Data Segregation / Multi-tenancy

[binayak (Binayak Das)]

Hi everyone, I’ve recently been exploring the Data Filter module while looking through the Integrating Data Filter for Data Segregation / Multi-tenancy project idea. Since most of my recent contributions have been around backend/service-layer logic in OpenMRS, this project seemed like a natural area for me to dive deeper into.

From my understanding so far, the Data Filter module uses Hibernate’s filtering mechanisms to inject additional conditions into queries so that data visibility can be restricted based on context — for example limiting patient data to locations a user has access to.

This seems like a powerful mechanism for implementing system-wide data segregation policies.

While reading through the project idea and some related code, I started thinking about a few areas that might be interesting to explore for extending the module.

Possible Areas of Exploration

1. Extensible Rule Definitions

Currently, filtering appears to be tied to predefined logic (such as location-based access). One possible direction could be making the filtering system more extensible, allowing administrators or modules to define additional rule templates such as:

• restricting observations by concept codes

• restricting data access based on user privileges

• applying location or program-based filters dynamically

This might allow implementations to define more granular policies without modifying core filtering logic.

---

2. Administrative Configuration Layer

Another area that seems interesting is introducing a configuration or administrative layer where associations between users, locations, or other contextual entities can be managed.

For example:

• mapping users or roles to specific location scopes

• configuring rule sets that apply across multiple modules

• defining which filters should be active in a given implementation

This could make the filtering system more flexible across different deployments.

---

3. Consistency Across O3 Modules

Since the O3 RefApp includes multiple frontend applications that interact with the backend services, it may also be worth considering how filtering behavior remains consistent across modules.

Ensuring that the same filtering rules apply uniformly at the service or DAO layer could help maintain predictable behavior regardless of which frontend component triggers the query.

---

Questions for the Community for better Understanding

As I’m still exploring the module and the broader design space, I’d really appreciate input from those who have worked with the Data Filter module or related access control mechanisms.

Some questions I’m particularly wanted to know about:

• What are the current limitations of the Data Filter module in real-world implementations?

• Are there known extension points or architectural constraints that should be considered when expanding its capabilities?

• Would it make sense to think about rule templates and configuration-driven filtering, or is the expectation that filtering logic remains more static?

I’m hoping to better understand how the community envisions this evolving so that any implementation directions I explore align well with real-world needs.

Looking forward to hearing thoughts from others who have worked in this area.

cc: @wyclif, @jnsereko, @ibacher, @dkayiwa, @mseaton, @mogoodrich, @wikumc

[binayak (Binayak Das)]

Hi everyone,

Just following up on this thread in case it got buried.

While reviewing the current approach using Hibernate filters, I started wondering about how extensible the filtering mechanism is intended to be across different modules in the O3 ecosystem.

For example, if implementations wanted to introduce additional filtering rules (such as concept-level restrictions, privilege-based rules, or program-specific filters), would the preferred direction be:

• extending the existing Hibernate filter definitions directly, or

• introducing a more configurable rule layer that determines which filters should be applied at runtime?

I’m trying to better understand how flexible the filtering architecture is expected to become as part of this project.

Any thoughts or pointers would be really helpful.

cc: @wyclif, @jnsereko, @ibacher, @dkayiwa, @mseaton, @mogoodrich, @wikumc

[mherman22 (herman muhereza)]

UgandaEMR: Use cases for the Data Filter module

Below are the steps i think need to be added to wiki. Qn. How to add a filter to the datafilter module without forking. Answer Add the datafilter module into your distro using maven like here exclude all the filters appended to the datafilter module by default (unless you need them). For example, see how all filters were excluded using global properties and the initializer Create a custom module and add the datafilter module like here make your custom module depend on the datafilter mo…

[binayak (Binayak Das)]

Thanks for sharing this UgandaEMR example @mherman22 which helped me a lot, going through that discussion helped to clarify how the Data Filter module is currently extended through implementation-specific modules without modifying the core module itself.

From what I understand so far, the typical pattern seems to be:

• include the Data Filter module in the distribution

• disable default filters if needed through configuration

• create a custom module that depends on the Data Filter module

• define and register implementation-specific filters there

This approach is flexible, but it also seems like each implementation needs to manually wire the filtering logic.

I’m wondering whether part of the improvement for this project could involve making this extension pattern more standardized — for example by introducing clearer rule templates or a configuration-driven way to register common filters (such as location-based, privilege-based, or concept-based filters).

Curious to hear from others who have implemented filters in real deployments:

• what kinds of filtering rules are most commonly needed?

• are there cases where the current extension approach becomes difficult to maintain or configure?

Any experiences from implementations would be really helpful to understand the practical needs here.

cc: @wyclif, @jnsereko, @mherman22, @ibacher, @dkayiwa, @mseaton, @mksd

[mherman22 (herman muhereza)]

@binayak

This approach is flexible, but it also seems like each implementation needs to manually wire the filtering logic.

Because each implementation has it’s own needs.

@binayak

configuration-driven way to register common filters (such as location-based, privilege-based, or concept-based filters).

Are you talking about something like this openmrs-module-datafilter/api/src/main/resources/filters at master · openmrs/openmrs-module-datafilter · GitHub? When you say a configuration driven way to register common filters?

[binayak (Binayak Das)]

Yes, @mherman22 that’s the part I was referring to.

My understanding is that the module already allows filters to be registered through configuration files under api/src/main/resources/filters, where the filter definitions (name, target classes, SQL condition, parameters) are declared.

However, it seems that implementations still need to manually implement the logic that provides the filter parameters (for example through custom DataFilterListener implementations), depending on how access rules are determined.

What I was thinking about is whether it might be useful to introduce higher-level reusable templates for common filtering patterns (for example location-based, privilege-based, or concept-based filters). Implementations could then configure these templates rather than writing custom listeners each time.

So the idea would be to build on top of the existing configuration-driven filter definitions, but make common patterns easier to reuse across implementations.

[mherman22 (herman muhereza)]

@binayak

location-based, privilege-based, or concept-based filters)

I still feel like this is an implementation detail.

However, a good starting point at the moment could be to upgrade the module to work with OpenMRS Core 2.8.x and above . The OpenMRS Core 2.8.x release introduced major changes that will require updates to the datafilter module. The most significant change for this module is the Hibernate Search 5.x to 6.x upgrade that OpenMRS Core 2.8.x introduced, which could require a rewrite of the full-text search filtering functionality. Among other things

[binayak (Binayak Das)]

Thanks for this @mherman22 which I was unaware of till now.

Upgrading the module to work with OpenMRS Core 2.8.x looks like a good starting point. I went through that the Hibernate Search upgrade from 5.x to 6.x introduced significant API changes, so the current full-text filtering implementation in the Data Filter module may need to be updated or rewritten to work with the new search APIs.

I’ll take a look at the parts of the module that integrate with full-text search (for example under the filters/fulltext area) to understand how the filtering is currently implemented and will comeback.

Just to know — would the main goal for the project would here be:

1. Ensuring the module is fully compatible with OpenMRS Core 2.8.x, and

2. Refactoring or rewriting the full-text search filtering to work with Hibernate Search 6?

Or are there other areas of the module that are known to be affected by the Core 2.8.x changes?

[mksd (Dimitri R)]

@binayak

• create a custom module that depends on the Data Filter module

Correct, but this was not the original intent. The initial idea was to provide the ability to inject filters through configuration… this was never developed in the end.

As the original developers of the module, at @Mekom we observed that the default filters already cover the most common use cases (which are generally related to location-based filtering.)

Our general stance is to avoid developing and proliferating custom modules, as we know too well for it to be a poor long-term practice. However, in some cases the data filtering requirements are so implementation-specific that it is reasonable for them to live in a custom module, as the exception that proves the rule

From our perspective as both the originators of the module and likely its largest users across implementations, we have not encountered many situations where changes to the module itself needed to be prioritized.

There may be some housekeeping to do though, @wyclif, your input welcome here. Maybe @frederic.deniger as well?

[jonathan (Mutagubya Jonathan)]

Hi @binayak Great to see you’re also exploring this module. I’m also very interested in participating in this project as a contributor and have been diving into the codebase to see how it currently ticks.

@binayak

1. Extensible Rule Definitions

Currently, filtering appears to be tied to predefined logic (such as location-based access). One possible direction could be making the filtering system more extensible, allowing administrators or modules to define additional rule templates such as:

• restricting observations by concept codes
• restricting data access based on user privileges
• applying location or program-based filters dynamically

This might allow implementations to define more granular policies without modifying core filtering logic.

Regarding your points, I think the “extensible rule definitions” are definitely the core of the project. We really need to expand the coverage to account for O3 esm modules. I’ve verified that domains like Allergy, Appointment, and Order aren’t currently filtered, yet they are central to the O3 RefApp clinical workflows.

The idea of rule templates is also huge. Specifically, being able to tie the visibility of Obs with specific concept codes to certain user privileges (e.g hiding HIV-related results from users without a 'View Sensitive Data' privilege) would be a game-changer for many implementations.

One thing I’m noticing is that the module is currently tied to Platform 2.4.x. If we want this to work smoothly with recent RefApp versions (Core 2.8+) just like @mherman22 has highlighted, the biggest technical hurdle might be the Hibernate Search 6 upgrade. Since the API there was basically a total rewrite, it’ll likely require overtaking the full-text search filtering logic in the module before we can properly expand it.

On the UI side, since the project lists Java and Hibernate as the main skills, I’m leaning towards focusing on a robust REST API layer and the rule engine first. That would give us a solid foundation to build out an O3 microfrontend later on as a community.

Curious to hear what the mentors @wyclif, @jnsereko … and @ibacher, @dkayiwa think about prioritizing the Hibernate Search 6 migration as a foundational step alongside the rule templates!

[binayak (Binayak Das)]

Thanks @mksd for this context, which also aligns what was said by @mherman22 in this discussion only.

From the discussion so far, it sounds like a good first step would be to focus on modernizing the module rather than introducing new functionality immediately which can be a follow-up. Based on the recent platform changes, the main objective could be to update the Data Filter module so that it works properly with newer OpenMRS Core versions.

A possible direction could be:

Core objective – Modernize the Data Filter module

• Make the module compatible with OpenMRS Core 2.8+

• Update the Hibernate Search integration (5 → 6)

• Refactor or rewrite the full-text filtering functionality where necessary

• Clean up parts of the architecture that rely on older APIs

• Improve documentation and extension points for implementers

Then as a secondary exploration, it might be interesting to revisit the original idea of allowing filters to be injected through configuration rather than requiring custom modules or listeners for common cases as the data-filter module is far behind as compared to be compatible with OpenMRS Core 2.8+ which I realised after going in deep with the module. But that would likely come after the modernization work.

While looking through the codebase (especially the filters/fulltext area and the session/listener integration), I noticed that the module currently hooks into Hibernate sessions and query creation in order to apply filters automatically.

For the Hibernate Search upgrade specifically, would the preferred approach be to adapt the current filtering mechanism to the new Hibernate Search 6 APIs, or would it make more sense to redesign how full-text filtering is applied (for example through the newer SearchSession / predicate APIs) while keeping the existing filter registration model?

Understanding that design direction would help guide how the migration should be approached.

Any thoughts. cc: @wyclif, @mksd, @ibacher, @dkayiwa, @mherman22

[binayak (Binayak Das)]

Hi @jonathan, it was great to see you also giving some insights over the topic but the main issue with going ahead with your architectural design is about the compatibility to work smoothly with recent RefApp versions (Core 2.8+) which you also mentioned but I think this should also be included in the project idea to start off with this and then we can go for the “extensible rule definitions” which is the core of the project as the migration will take a lot of effort and time which is achievable with the gsoc idea timeline.

I agree that expanding the filtering coverage for O3 domains like Allergy, Appointment, and Order would be very valuable, especially since those entities are becoming central in O3 workflows.

While looking into the module internals, one thing I noticed is that the current architecture relies heavily on intercepting Hibernate sessions through DataFilterSessionContext, where filters are dynamically enabled based on registered DataFilterListeners. That works well for standard Hibernate queries, but it seems that full-text queries are handled separately through the FullTextQueryCreatedEventListener.

Given that OpenMRS Core 2.8 upgraded Hibernate Search from 5 → 6 (with a significantly different API), it seems likely that the full-text filtering part of the module may need a substantial rewrite before other improvements can be layered on top.

One design question I’m wanting to clarify:

Since the module currently applies filters transparently at the Hibernate session level, would it still make sense to keep that same session-level interception model when adapting to Hibernate Search 6, or would a query-level integration (e.g. building predicates via the new SearchSession APIs) be the preferred direction going forward?

Any thoughts. cc: @jonathan, @mherman22, @mksd, @wyclif, @ibacher, @dkayiwa

[mksd (Dimitri R)]

@binayak

Core objective – Modernize the Data Filter module

• Make the module compatible with OpenMRS Core 2.8+
• Update the Hibernate Search integration (5 → 6)
• Refactor or rewrite the full-text filtering functionality where necessary
• Clean up parts of the architecture that rely on older APIs
• Improve documentation and extension points for implementers

This is definitely awesome as a roadmap First and foremost the compatibility with the most recent versions of Core.

[binayak (Binayak Das)]

Yeah…thanks for guidance @mksd !