Sync 2.0 Planning

sync2
Tags: #<Tag:0x00007f295606ab68>

(Paweł Gesek) #1

Hello everyone,

@raff and I are happy to announce that SolDevelo will be starting work on the Sync 2.0 project, which will aim to be a FHIR based replacement for the existing Sync 1.0 module. We are excited and looking forward towards starting work on this project, as it aims to solve real issues that OpenMRS implementations have faced when using the current version of Sync.

We are looking for implementations that are interested in using the new Sync module and willing to commit to early adoption and testing of the module. We are also interested to hear from anyone that has used Sync 1.0 about their workflows and the challenges they faced when using it, so that we can solve these issues with the design for version 2.0. If you are in any way interested in Sync 2.0, feel free to reach out to us in this thread!

Sync 2.0 Project Description


Initial Statement: Replacement for openmrs sync module based on FHIR (as much as possible) and atom feeds - LOE: 6+ months - Customers: of interest to many existing OpenMRS implementations (many are desperate for a usable & reliable sync solution)

Specific Project: Support master-slave synchronization of multiple OpenMRS servers in one enterprise, where most (technical) management is done at a central site, but patients are seen at many clinics in a hospital network.

Technical Approach:

  • one Master OpenMRS server, and multiple Slave OpenMRS servers.
  • Master defines all metadata and configuration (and slaves synchronize this).
  • Slaves synchronize a subset of patients with Master (e.g. those in the catchment area of one clinic) and push data on these patients
  • Slaves initiate all synchronization, based on:
    • reading an atom feed published by Master, pulling more data for any events of interest
    • pushing data that is entered on the Slave

Project Breakdown:

There are several independently-useful parts of this project that can be done before pulling everything together.

  1. Publish an event feed from OpenMRS
  • can leverage existing ICT4H and Bahmni code for this
  1. Update the OpenMRS FHIR module
    • Update from DSTU2 to STU3 (released April 2017)
    • Refactor module to be more extensible * support for plugging in processors & resources * use a Strategy Pattern, and extract current hardcoded implementations as “default strategy” * Ensure we can import/export all relevant OpenMRS transactional data
  2. Implement one-way sync of domain objects that can’t be fully represented using FHIR (e.g. some kinds of metadata)
    • ThoughtWorks wrote a module for Bangladesh that synchronizes concepts; may be ready to use as-is
  3. Mechanism to define patient subsets / catchment areas for subcenters We already know of some implementation strategies from existing research done by Bahmni More requirements gathering to see if those strategies miss any high-value use case
  4. Pull all of this together into the Sync solution

Thanks, Paweł


Capability Questions
Sync 2.0 Discussion about pull/push architecture
Moving Patient data from one OpenMRS installation to another
Welcome! Please introduce yourself.
Propose a Design Forum Topic
Request for new repositories for Sync 2 module and atomfeed
Welcome! Please introduce yourself.
(Ellen Ball) #2

Partners In Health has been using Sync V1.0 in Rwanda, Malawi, and Haiti for many years. We will pull together workflow and challenges to share.

This is excellent news!

Ellen


(Dimitri R) #3

Hi @pgesek and Soldevelo team, this is very interesting and we will be following this development eagerly.

@craigappl, how does this relate to your MPI sync effort that has been discussed on yesterday’s design forum?


(Stephen Senkomago Musoke) #4

Would Sync 2.0 also provide a mechanism for automatically pulling data into OpenMRS from 3rd party systems like Lab etc?

Thinking aloud even more can the functionality also be used to automate pushing data to another system not necessarily OpenMRS?


(Craig Appl) #5

Hi Everyone,

I would like to support the design phase of this work. We had a really great design call yesterday on integration with an MPI and I’m working to see how these two initiatives complement each other.

I think it would be good to define assumptions in each:

Sync 2.0 (based on the project description above)

  • There is a single enterprise owner for all OpenMRS servers. Cross organization sync is out of scope.
  • The project focuses on OpenMRS to OpenMRS data sync, not sync with third party systems
  • FHIR is the preferred method of interaction and one-way sync will be used only where FHIR isn’t capable
  • The lower-level clinic (slave) is the primary actor for initializing synchronization
  • I think the description assumes 100% sync of all metadata between the master and slave and 100% of patients that are defined in the patient subsets in bullet 4 of the Project Breakdown.

PIX/PDQ Integration with an MPI

  • Assumes different enterprise owners between the MPI and the OpenMRS implementation
  • Solely focuses on sharing registration information (Query MPI, Post new registrations, Post demographic information updates)
  • Utilizes the standards defined in IHE’s PIX(m)/PDQ(m) profiles so any MPI can be used
  • Assumes the OpenMRS clinic is the primary initiator of the interaction with the MPI
  • (Working Assumption) Every change to the patient’s registration record should be pushed to the MPI

FYI: @ssmusoke, @mksd


(Darius Jazayeri) #6

@craigappl’s breakdown is exactly right.

(Just to clarify/confirm, in Sync each child would have a subset of the patients in the parent, but 100% of the child’s patients would be syncd.)

At a project level we should approach this from the lens of “Sync 2.0”. But at a technical architecture level we should build several independently-useful pieces that can be used outside of Sync.

E.g. we’ll build a core-supported way of publishing event feeds, and we’ll improve the functionality of the FHIR module.

I suggested on yesterday’s call that the MPI integration project should use the same event feed module as Sync 2.0, rather than building its own (or using the event module).

As Craig said, “Cross organization sync is out of scope.” (and, cross-system sync is also out of scope).

Some of the building blocks of sync could be leveraged to help do these things, though. E.g. leverage the “poll the event feed” and “transform events into FHIR” that are built in this project, but you’d still have to build the custom “push data to another system”.


(Stephen Senkomago Musoke) #7

@darius @craigappl while cross-organization and cross-sync may be out of scope, can the architecture/design/implementation for OpenMRS servers be that the processors/consumers/subscribers that are developed.

This means that implementing other connectors/use cases is just work but already architecturally supported.


(Paweł Gesek) #8

I agree that we should think about having the architecture generic with the synchronization connector/logic pluggable. That way Sync 2.0 could be used to synchronize into a central HIE infrastructure (synchronize patients with an MPI like OpenEMPI) for example, instead of a master OpenMRS instance - this is a use case me & @craigappl are interested in, more on this in Integration with OpenEMPI.

Implementing such a connector would require additional effort obviously, but as Darius mentioned, components such as FHIR based communication or the event feed could be reused in this case.


(Mike Seaton) #9

I am very excited to see this project taken up by @SolDevelo and the community. I think this has the potential to be a transformational project, and one which PIH is sure to want to leverage in many of our implementations. I am also very supportive of the planned approach, to build several independently useful features - many of which would be huge advances in their own right. Looking forward to helping to make this a success.

Mike


(Darius Jazayeri) #10

I enjoy the conversation thread so far, but let me highlight this important question. We are looking for 1-2 implementations to commit to being early adopters and testers of this new sync approach, which will be built out over ~6 months.

@pgesek, maybe you want to cross-post on the #implementing category and ask that specific question.


(Ada Yeung) #11

Hi, @darius, on behalf of FGH Mozambique (@eurico.jose @steliomo @emaposse @damasceno_lopes @willa @matcha @ayeung), we would like to sign up for testing.


(Craig Appl) #12

@darius, @pgesek

Do we expect Sync 2.0 to sync every database transaction both ways, or the most recent snapshot at the time of the sync? I’m envisioning a scenario where a clinic is offline for a significant period of time. A patient’s demographic information has been updated three times in the person table and the atom feed shows that three transactions have happened. The atom feed will query the REST API three times, but do they download the changes that happened each of the three times, or just the most recent person record?

The same thing goes for the push from the slave clinic. Do we expect to push each database transaction that happens on the patient’s record, or do we expect to locally query the REST API when the sync transaction is scheduled to run?

Thanks, Craig


(Darius Jazayeri) #13

The current sync module captures all of the individual deltas, and synchronizes them in-order. This has some advantages, but it ties everything very closely to a specific version of the OpenMRS data model, and has made things quite brittle.

In Sync 2.0 we should take the other approach, and synchronize the complete most recent version of the record. FHIR has a Patient resource but not a PatientDemographicEdit resource. This also implies that if the demographics were edited 3 times while a slave was offline, it only needs to pull the patient once after coming back online.

(This is just my opinion, before any deep technical analysis, and I’m not the one who will be leading that; but I do feel pretty strongly about it!)


(Burke Mamlin) #14

+1. It’s worth picturing how implementations could evolve from Sync 2.0 into “enterprise sync” (i.e., HIE). Taking a higher level “give me the latest info for this patient” approach is closer to the types of transactions used in an HIE.

@pgesek, you can refer to the following post for instructions on how to create a linked topic in OpenMRS Talk:


(Craig Appl) #15

Thanks @darius,

@pgesek & @raff Along those lines, I expect we would want some way to expose a patient-level transaction history to users of the slave systems so they can view how an individual patient record changed over time, in the event that a patient shows up in their clinic after their record has been augmented by a feed read and sync from the master.

We will also want to consider robust auditing of all transactions in the master system and exposing those to users anywhere in the enterprise.

Craig


(Todd Anderson) #16

Here in Partners in Health, Rwanda, Sync 1.0 has been central to our service since ~2011. I’ll summarize here a bit of our experience. Advancement of this functionality comes up continually for us both for our own sites (~45) and across the country of Rwanda.

Usage We have two sync networks, one with 25 children and the other with 20. These networks sync our modules, configurations, and html forms as well as patient data. Currently, patient data is synced up to the central server and then that same data is snyced down to each server.

The servers are connected to each other either through the internet or a cellular modem-based APN connecting the rural servers.

Sync has been central to our ability to capture information from our clinical programs and do central reporting. However, it has been very costly in maintenance time to do so.

Experience Network issues In Rwanda, we have significant issue with uptime to our rural servers trying to connect but only doing so intermittently. This is an even larger issue when these child servers must be accessed for certain data or fixes.

Errors Historically, the largest problem with Sync 1.0 had been the number of sync errors. Each sync error completely stopped the serial transmission of data in that direction from that server. These then had to be fixed by one of our developers–a particularly difficult task on difficult-to-access rural servers.

Size To keep the amount of data sent more reasonable, we partitioned the network into two parts. We would prefer to have a single network and the country of Rwanda would like to be able to capture information from around the whole country.

From time to time we simply bring all the servers to a central point to do more fundamental updates and to get an updated sync.

Echoing Ellen and Mike, improved sync would be of great value to us here in PIH-Rwanda and we’d be happy to share more of our experience.


(Alejandro Ramirez) #17

Hello Global Brigades would like to be part of this process.

We are currently running OpenMRS 1.9.8 and Sync 1.3 on our Servers:

We have 3 country nodes (In 2 separate continents:Africa,America), with one main server(Cloud). The Nodes consist of 10,11 and 1 field servers.

We currently have little issues besides certain registries sometimes getting stuck and of course the Very Long time it takes the sync process for large numbers of registries.


(Jamie Thomas) #18

@pgesek and @raff it seems like you all might benefit from some design time for Sync 2.0. I would love to get something scheduled for everyone to talk. Can you reply post your topic to:


(Rafal Korytkowski) #19

@jthomas, thanks! We’ll definitely ask for time on a design call at some point. First, we need to write down what was discussed here in an organized form in a wiki.

@mseaton, @mogoodrich, @PIH, since Sync 1.x is under your wings, would you be okay, if we used sync github repo (a new 2.x branch for now), wiki space and JIRA project for the work on Sync 2.0?

Thank you all for input. It is very helpful. We’ll try to gather all we learned and share it soon.


(Mike Seaton) #20

@raff, I think that makes sense, assuming that’s the intention of this work - to be the next version of the sync module…