Sync 2.0 Planning

sync2
Tags: #<Tag:0x00007f23d8ee9528>

(Darius Jazayeri) #13

The current sync module captures all of the individual deltas, and synchronizes them in-order. This has some advantages, but it ties everything very closely to a specific version of the OpenMRS data model, and has made things quite brittle.

In Sync 2.0 we should take the other approach, and synchronize the complete most recent version of the record. FHIR has a Patient resource but not a PatientDemographicEdit resource. This also implies that if the demographics were edited 3 times while a slave was offline, it only needs to pull the patient once after coming back online.

(This is just my opinion, before any deep technical analysis, and I’m not the one who will be leading that; but I do feel pretty strongly about it!)


(Burke Mamlin) #14

+1. It’s worth picturing how implementations could evolve from Sync 2.0 into “enterprise sync” (i.e., HIE). Taking a higher level “give me the latest info for this patient” approach is closer to the types of transactions used in an HIE.

@pgesek, you can refer to the following post for instructions on how to create a linked topic in OpenMRS Talk:


(Craig Appl) #15

Thanks @darius,

@pgesek & @raff Along those lines, I expect we would want some way to expose a patient-level transaction history to users of the slave systems so they can view how an individual patient record changed over time, in the event that a patient shows up in their clinic after their record has been augmented by a feed read and sync from the master.

We will also want to consider robust auditing of all transactions in the master system and exposing those to users anywhere in the enterprise.

Craig


(Todd Anderson) #16

Here in Partners in Health, Rwanda, Sync 1.0 has been central to our service since ~2011. I’ll summarize here a bit of our experience. Advancement of this functionality comes up continually for us both for our own sites (~45) and across the country of Rwanda.

Usage We have two sync networks, one with 25 children and the other with 20. These networks sync our modules, configurations, and html forms as well as patient data. Currently, patient data is synced up to the central server and then that same data is snyced down to each server.

The servers are connected to each other either through the internet or a cellular modem-based APN connecting the rural servers.

Sync has been central to our ability to capture information from our clinical programs and do central reporting. However, it has been very costly in maintenance time to do so.

Experience Network issues In Rwanda, we have significant issue with uptime to our rural servers trying to connect but only doing so intermittently. This is an even larger issue when these child servers must be accessed for certain data or fixes.

Errors Historically, the largest problem with Sync 1.0 had been the number of sync errors. Each sync error completely stopped the serial transmission of data in that direction from that server. These then had to be fixed by one of our developers–a particularly difficult task on difficult-to-access rural servers.

Size To keep the amount of data sent more reasonable, we partitioned the network into two parts. We would prefer to have a single network and the country of Rwanda would like to be able to capture information from around the whole country.

From time to time we simply bring all the servers to a central point to do more fundamental updates and to get an updated sync.

Echoing Ellen and Mike, improved sync would be of great value to us here in PIH-Rwanda and we’d be happy to share more of our experience.


(Alejandro Ramirez) #17

Hello Global Brigades would like to be part of this process.

We are currently running OpenMRS 1.9.8 and Sync 1.3 on our Servers:

We have 3 country nodes (In 2 separate continents:Africa,America), with one main server(Cloud). The Nodes consist of 10,11 and 1 field servers.

We currently have little issues besides certain registries sometimes getting stuck and of course the Very Long time it takes the sync process for large numbers of registries.


(Jamie Thomas) #18

@pgesek and @raff it seems like you all might benefit from some design time for Sync 2.0. I would love to get something scheduled for everyone to talk. Can you reply post your topic to:


(Rafal Korytkowski) #19

@jthomas, thanks! We’ll definitely ask for time on a design call at some point. First, we need to write down what was discussed here in an organized form in a wiki.

@mseaton, @mogoodrich, @PIH, since Sync 1.x is under your wings, would you be okay, if we used sync github repo (a new 2.x branch for now), wiki space and JIRA project for the work on Sync 2.0?

Thank you all for input. It is very helpful. We’ll try to gather all we learned and share it soon.


(Mike Seaton) #20

@raff, I think that makes sense, assuming that’s the intention of this work - to be the next version of the sync module…


(Paweł Gesek) #21

Thanks @ayeung and @aramirez - we are looking forward to working with FGH Mozambique & Global Brigades during the development of Sync 2.0. Your testing and input on the design/development of Sync 2.0 will definitely help us build a solution that will meet the needs of the community.

@raff and @jthomas - I agree that the next step will be documenting on the wiki (in the Sync space) and organizing a design call to discuss further.


(Darius Jazayeri) #22

I would also consider the option that although this will be a replacement for the sync module, we might choose to give it a new name. And since the codebase will be 100% new there is no clear value to doing this in the old sync module’s git repo.

-Darius (by phone)


(Mark Goodrich) #23

(thought I posted this in response to Mike, but just seeing now that I didn’t… )

Since it is going to be a completely new code base, doesn’t it make sense to start a new repo? I guess from a naming convention it does make it easier. Is there any kind of best practice around this type of situation?

That being said, if the general consensus is to use the same repo, I’m fine with that…

Mark


(Wyclif Luyima) #24

I’d also vote for having the new version in a new repo since the codebase is really completely different, what would be the module id though?


(Darius Jazayeri) #25

Angular 1 and 2 are in two different repos:

Jackson JSON 1 is in svn and 2 is in github:

OpenLMIS created a new set of repos when they did a complete refactor/rewrite (I advised on this):

So, it seems like large-scale code rewrites prefer to create new repos. (E.g. you would never want a developer to do a git pull and inadvertently go from sync 1.x to sync 2.x.)

I recommend that we delay on deciding the module id and the exact naming/marketing as we do design, figure out what different independent components are involved, what might overlap with a connect-to-MPI/HIE project, etc. (We can continue to call the project Sync 2.0 in the interim.)


(Erick Wafula) #26

KEMRI-SEARCH and KEMRI-FACES are extensive users of sync 1.x version and I would be happy to be part of the testing team.

Erick


(Craig Appl) #27

FYI @carl I want to make sure you’re informed about this conversation because you’re doing so much work with HIE and FHIR.


(Paweł Gesek) #28

Thanks @werick, we will keep you in the loop.

I don’t see an issue with doing a new git repo - it has some advantages over branching.

Did we decide on using the existing Jira and Wiki? I noticed that there are about 50 unresolved issues for sync - doesn’t necessarily mean we need a new project, we can just use the version fields to keep track of that.

If we are using the existing sync wiki, I can start putting together a doc based on the initial requirements and discussions on talk.


(Ryan Crichton) #29

This sounds like a great effort. We are rather invested in FHIR so this seems like a excellent move.

It may be worth considering the FHIR transaction/batch interactions to perform the sync in a push model. Otherwise, looking at performing global searches for a pull model.

I don’t believe FHIR makes use of ATOM feeds anymore, rather they have moved to using a Bundle resource to list multiple resources along with metadata.


(Rafal Korytkowski) #30

We can start documenting at https://wiki.openmrs.org/display/projects/Sync+2.0

Typically a JIRA project is linked with one repo so not sure, if we should reuse the JIRA project, but not the repo, unless the idea is to retire the old repo.


(Paweł Gesek) #31

Hello everyone,

I have added design documentation for the module as child pages under:

https://wiki.openmrs.org/display/projects/Sync+2.0

Please let me know any feedback you have - I will be incorporating it in the documentation.

Thanks, Paweł


Help define the short-term Bahmni Roadmap
(Darius Jazayeri) #32

Thanks @pgesek I was just wondering whether there was an update on this. :slight_smile:

Comments about “Atomfeed for Sync 2.0”:

  • If we’re going to refactor this, maybe we can move away from having a lot of global properties, and instead just have a single feed config that an implementation can specify with a json or yaml file
  • I like the idea of using an event-based hibernate interceptor instead of an advice class for every OpenMRS class
  • “Change the default urls to point to FHIR resources” => I would hope that Sync 2.0 can also be run with Bahmni, so I hope there’s a way that a single set of feeds can work for both master-slave sync, and Bahmni’s MRS/ERP/LIS communication. Maybe we can introduce a new format where we provide multiple possible links to the event, e.g. both the FHIR one and the REST one.

Comments about “Auditing and error handling”:

  • I wonder if this is going to be an excessive amount of data to be storing
  • It feels more natural to log this to a file (at least for successful sync) rather than to a DB table
  • PIH Rwanda found that for admin/management purposes it’s really valuable to synchronize status data about each slave up to the master, so that some level of system debugging can be done centrally

“Catchment Strategies”:

  • I guess we should get some feedback from potential implementations about what are the right strategies to focus on

“FHIR Resource List for Sync”

  • Naively I would think Drug => Medication (instead of Substance)
  • Some of these may be very imperfect matches, and after analysis we might end up wanting to do some of these via OpenMRS REST representations.
  • There’s definitely a prioritization to be done here. E.g. some implementation might start using sync with just patient. Others would use it if you add visit, encounter, and obs. Etc.

(I will try to comment on the rest of the pages tomorrow, but I’ll post this now, in case I get delayed on the rest.)