Sync 2.0 Planning

Hi @pgesek and Soldevelo team, this is very interesting and we will be following this development eagerly.

@craigappl, how does this relate to your MPI sync effort that has been discussed on yesterday’s design forum?

Would Sync 2.0 also provide a mechanism for automatically pulling data into OpenMRS from 3rd party systems like Lab etc?

Thinking aloud even more can the functionality also be used to automate pushing data to another system not necessarily OpenMRS?

Hi Everyone,

I would like to support the design phase of this work. We had a really great design call yesterday on integration with an MPI and I’m working to see how these two initiatives complement each other.

I think it would be good to define assumptions in each:

Sync 2.0 (based on the project description above)

  • There is a single enterprise owner for all OpenMRS servers. Cross organization sync is out of scope.
  • The project focuses on OpenMRS to OpenMRS data sync, not sync with third party systems
  • FHIR is the preferred method of interaction and one-way sync will be used only where FHIR isn’t capable
  • The lower-level clinic (slave) is the primary actor for initializing synchronization
  • I think the description assumes 100% sync of all metadata between the master and slave and 100% of patients that are defined in the patient subsets in bullet 4 of the Project Breakdown.

PIX/PDQ Integration with an MPI

  • Assumes different enterprise owners between the MPI and the OpenMRS implementation
  • Solely focuses on sharing registration information (Query MPI, Post new registrations, Post demographic information updates)
  • Utilizes the standards defined in IHE’s PIX(m)/PDQ(m) profiles so any MPI can be used
  • Assumes the OpenMRS clinic is the primary initiator of the interaction with the MPI
  • (Working Assumption) Every change to the patient’s registration record should be pushed to the MPI

FYI: @ssmusoke, @mksd

1 Like

@craigappl’s breakdown is exactly right.

(Just to clarify/confirm, in Sync each child would have a subset of the patients in the parent, but 100% of the child’s patients would be syncd.)

At a project level we should approach this from the lens of “Sync 2.0”. But at a technical architecture level we should build several independently-useful pieces that can be used outside of Sync.

E.g. we’ll build a core-supported way of publishing event feeds, and we’ll improve the functionality of the FHIR module.

I suggested on yesterday’s call that the MPI integration project should use the same event feed module as Sync 2.0, rather than building its own (or using the event module).

As Craig said, “Cross organization sync is out of scope.” (and, cross-system sync is also out of scope).

Some of the building blocks of sync could be leveraged to help do these things, though. E.g. leverage the “poll the event feed” and “transform events into FHIR” that are built in this project, but you’d still have to build the custom “push data to another system”.

1 Like

@darius @craigappl while cross-organization and cross-sync may be out of scope, can the architecture/design/implementation for OpenMRS servers be that the processors/consumers/subscribers that are developed.

This means that implementing other connectors/use cases is just work but already architecturally supported.

I agree that we should think about having the architecture generic with the synchronization connector/logic pluggable. That way Sync 2.0 could be used to synchronize into a central HIE infrastructure (synchronize patients with an MPI like OpenEMPI) for example, instead of a master OpenMRS instance - this is a use case me & @craigappl are interested in, more on this in Integration with OpenEMPI.

Implementing such a connector would require additional effort obviously, but as Darius mentioned, components such as FHIR based communication or the event feed could be reused in this case.

I am very excited to see this project taken up by @SolDevelo and the community. I think this has the potential to be a transformational project, and one which PIH is sure to want to leverage in many of our implementations. I am also very supportive of the planned approach, to build several independently useful features - many of which would be huge advances in their own right. Looking forward to helping to make this a success.

Mike

I enjoy the conversation thread so far, but let me highlight this important question. We are looking for 1-2 implementations to commit to being early adopters and testers of this new sync approach, which will be built out over ~6 months.

@pgesek, maybe you want to cross-post on the Implementing category and ask that specific question.

Hi, @darius, on behalf of FGH Mozambique (@eurico.jose @steliomo @emaposse @damasceno_lopes @willa @matcha @ayeung), we would like to sign up for testing.

@darius, @pgesek

Do we expect Sync 2.0 to sync every database transaction both ways, or the most recent snapshot at the time of the sync? I’m envisioning a scenario where a clinic is offline for a significant period of time. A patient’s demographic information has been updated three times in the person table and the atom feed shows that three transactions have happened. The atom feed will query the REST API three times, but do they download the changes that happened each of the three times, or just the most recent person record?

The same thing goes for the push from the slave clinic. Do we expect to push each database transaction that happens on the patient’s record, or do we expect to locally query the REST API when the sync transaction is scheduled to run?

Thanks, Craig

The current sync module captures all of the individual deltas, and synchronizes them in-order. This has some advantages, but it ties everything very closely to a specific version of the OpenMRS data model, and has made things quite brittle.

In Sync 2.0 we should take the other approach, and synchronize the complete most recent version of the record. FHIR has a Patient resource but not a PatientDemographicEdit resource. This also implies that if the demographics were edited 3 times while a slave was offline, it only needs to pull the patient once after coming back online.

(This is just my opinion, before any deep technical analysis, and I’m not the one who will be leading that; but I do feel pretty strongly about it!)

1 Like

+1. It’s worth picturing how implementations could evolve from Sync 2.0 into “enterprise sync” (i.e., HIE). Taking a higher level “give me the latest info for this patient” approach is closer to the types of transactions used in an HIE.

@pgesek, you can refer to the following post for instructions on how to create a linked topic in OpenMRS Talk:

Thanks @darius,

@pgesek & @raff Along those lines, I expect we would want some way to expose a patient-level transaction history to users of the slave systems so they can view how an individual patient record changed over time, in the event that a patient shows up in their clinic after their record has been augmented by a feed read and sync from the master.

We will also want to consider robust auditing of all transactions in the master system and exposing those to users anywhere in the enterprise.

Craig

Here in Partners in Health, Rwanda, Sync 1.0 has been central to our service since ~2011. I’ll summarize here a bit of our experience. Advancement of this functionality comes up continually for us both for our own sites (~45) and across the country of Rwanda.

Usage We have two sync networks, one with 25 children and the other with 20. These networks sync our modules, configurations, and html forms as well as patient data. Currently, patient data is synced up to the central server and then that same data is snyced down to each server.

The servers are connected to each other either through the internet or a cellular modem-based APN connecting the rural servers.

Sync has been central to our ability to capture information from our clinical programs and do central reporting. However, it has been very costly in maintenance time to do so.

Experience Network issues In Rwanda, we have significant issue with uptime to our rural servers trying to connect but only doing so intermittently. This is an even larger issue when these child servers must be accessed for certain data or fixes.

Errors Historically, the largest problem with Sync 1.0 had been the number of sync errors. Each sync error completely stopped the serial transmission of data in that direction from that server. These then had to be fixed by one of our developers–a particularly difficult task on difficult-to-access rural servers.

Size To keep the amount of data sent more reasonable, we partitioned the network into two parts. We would prefer to have a single network and the country of Rwanda would like to be able to capture information from around the whole country.

From time to time we simply bring all the servers to a central point to do more fundamental updates and to get an updated sync.

Echoing Ellen and Mike, improved sync would be of great value to us here in PIH-Rwanda and we’d be happy to share more of our experience.

Hello Global Brigades would like to be part of this process.

We are currently running OpenMRS 1.9.8 and Sync 1.3 on our Servers:

We have 3 country nodes (In 2 separate continents:Africa,America), with one main server(Cloud). The Nodes consist of 10,11 and 1 field servers.

We currently have little issues besides certain registries sometimes getting stuck and of course the Very Long time it takes the sync process for large numbers of registries.

1 Like

@pgesek and @raff it seems like you all might benefit from some design time for Sync 2.0. I would love to get something scheduled for everyone to talk. Can you reply post your topic to:

@jthomas, thanks! We’ll definitely ask for time on a design call at some point. First, we need to write down what was discussed here in an organized form in a wiki.

@mseaton, @mogoodrich, @PIH, since Sync 1.x is under your wings, would you be okay, if we used sync github repo (a new 2.x branch for now), wiki space and JIRA project for the work on Sync 2.0?

Thank you all for input. It is very helpful. We’ll try to gather all we learned and share it soon.

1 Like

@raff, I think that makes sense, assuming that’s the intention of this work - to be the next version of the sync module…

Thanks @ayeung and @aramirez - we are looking forward to working with FGH Mozambique & Global Brigades during the development of Sync 2.0. Your testing and input on the design/development of Sync 2.0 will definitely help us build a solution that will meet the needs of the community.

@raff and @jthomas - I agree that the next step will be documenting on the wiki (in the Sync space) and organizing a design call to discuss further.

I would also consider the option that although this will be a replacement for the sync module, we might choose to give it a new name. And since the codebase will be 100% new there is no clear value to doing this in the old sync module’s git repo.

-Darius (by phone)