Thanks @ayeung and @aramirez - we are looking forward to working with FGH Mozambique & Global Brigades during the development of Sync 2.0. Your testing and input on the design/development of Sync 2.0 will definitely help us build a solution that will meet the needs of the community.
@raff and @jthomas - I agree that the next step will be documenting on the wiki (in the Sync space) and organizing a design call to discuss further.
I would also consider the option that although this will be a replacement for the sync module, we might choose to give it a new name. And since the codebase will be 100% new there is no clear value to doing this in the old sync moduleās git repo.
-Darius (by phone)
(thought I posted this in response to Mike, but just seeing now that I didnātā¦ )
Since it is going to be a completely new code base, doesnāt it make sense to start a new repo? I guess from a naming convention it does make it easier. Is there any kind of best practice around this type of situation?
That being said, if the general consensus is to use the same repo, Iām fine with thatā¦
Mark
Iād also vote for having the new version in a new repo since the codebase is really completely different, what would be the module id though?
Angular 1 and 2 are in two different repos:
Jackson JSON 1 is in svn and 2 is in github:
OpenLMIS created a new set of repos when they did a complete refactor/rewrite (I advised on this):
So, it seems like large-scale code rewrites prefer to create new repos. (E.g. you would never want a developer to do a git pull and inadvertently go from sync 1.x to sync 2.x.)
I recommend that we delay on deciding the module id and the exact naming/marketing as we do design, figure out what different independent components are involved, what might overlap with a connect-to-MPI/HIE project, etc. (We can continue to call the project Sync 2.0 in the interim.)
KEMRI-SEARCH and KEMRI-FACES are extensive users of sync 1.x version and I would be happy to be part of the testing team.
Erick
FYI @carl I want to make sure youāre informed about this conversation because youāre doing so much work with HIE and FHIR.
Thanks @werick, we will keep you in the loop.
I donāt see an issue with doing a new git repo - it has some advantages over branching.
Did we decide on using the existing Jira and Wiki? I noticed that there are about 50 unresolved issues for sync - doesnāt necessarily mean we need a new project, we can just use the version fields to keep track of that.
If we are using the existing sync wiki, I can start putting together a doc based on the initial requirements and discussions on talk.
This sounds like a great effort. We are rather invested in FHIR so this seems like a excellent move.
It may be worth considering the FHIR transaction/batch interactions to perform the sync in a push model. Otherwise, looking at performing global searches for a pull model.
I donāt believe FHIR makes use of ATOM feeds anymore, rather they have moved to using a Bundle resource to list multiple resources along with metadata.
We can start documenting at Sync 2.0 - Projects - OpenMRS Wiki
Typically a JIRA project is linked with one repo so not sure, if we should reuse the JIRA project, but not the repo, unless the idea is to retire the old repo.
Hello everyone,
I have added design documentation for the module as child pages under:
https://wiki.openmrs.org/display/projects/Sync+2.0
Please let me know any feedback you have - I will be incorporating it in the documentation.
Thanks, PaweÅ
Thanks @pgesek I was just wondering whether there was an update on this. 
Comments about āAtomfeed for Sync 2.0ā:
Comments about āAuditing and error handlingā:
āCatchment Strategiesā:
āFHIR Resource List for Syncā
patient. Others would use it if you add visit, encounter, and obs. Etc.(I will try to comment on the rest of the pages tomorrow, but Iāll post this now, in case I get delayed on the rest.)
Nice to see this going. few comments
In Bangladesh, we did all these for integrating with an HIE and using SimpleFeed as a protocol for event notification! It has served the purpose very well, but we needed to do a whole lotta work to ensure tackling problems (including ongoing support, monitoring) - many of which are relevant to out-of-order-event processing.
Why I say this repeatedly, I think that there can not be a singular subscribed mechanism for addressing such distributed system synchronization needs. All you can have is a broad framework that will probably deal with 80% of the cases, while allowing for extensibility for customization. If we fail to accommodate for the extensibility, then it will not serve general purpose. (it will serve specific purpose very well for sure)
@pgesek, thanks for pushing the project forward!
Do you have some time aside in the coming days to finalize planning and/or any estimate when development could start?
Iām less available these days due to another project Iām involved in, but will help as much as I can. Let me know, if there are any gaps, which I could help you fill in.
Thanks for doing all this @pgesek!
Iāve tried to skim through everything, a few pieces of feedback:
+1 to using Hibernate Interceptors instead of AOP to figure out what to publish to the feed, as AOP is way to brittle. Maybe Iām forgetting, but is there a reason we wouldnāt be using the existing Event module for for this? @raff @darius
The feed mechanism should be robust enough to handle a slave being down on the order of days, or perhaps even months
Darius may be right that storing all the audit information to a DB table might be excessive, but from experience debugging Sync 1.0 issues, I do feel that having good admin tools for resolving issues, etc, will be critical, so Iād err on the side of whatever design you feel will allow these tools to be easiest to design, develop, and tweak.
I see @angshuonline point about āmaster/slaveā not being the correct terminology (https://en.wikipedia.org/wiki/Master/slave_(technology)). Not sure if I like client/server though as it might not be clear enough. Is there a reason not to use parent/child, as was used in the initial Sync module?
Take care, Mark
Thanks for the responses, my replies:
Yes, that makes sense and should be cleaner, Iāll update the feed doc.
An option is to use multiple link tags. Would be best imo to include the url configuration into this new atomfeed configuration file.
Yes, error handling is probably the biggest gripe users have with Sync 1.0, hence the the amount of logging. I agree that the success logs are probably not as important as the failure ones - we can log success messages to a file by default. Perhaps it would be best to allow users to configure which logger implementation to use for event types like success and failure. Sync could provide some default implementations like DB, File, No-op and implementers could inject their own if need be.
Iāve done some basic ordering of the list, Iāve also added a FHIR maturity level column.
On that page I listed the catchment strategies I got from implementers - feel free to add any additional ones.
Yes, my bad.
My hope is that on the sending/receiving end, we will have strategies that should allow controlling the mapping. We should also make it always possible for an implementation to inject their own client implementation for a given OpenMRS class (FHIR or not). It should also be possible to register additional providers on the receiving side in the FHIR module. Iāll start an extension section in the FHIR doc to gather information on extending how it works.
I donāt have a strong opinion on the terminology, I used what was in the project description. If Sync 1.0 uses parent/child then perhaps itās best to stick to it.
Thanks, Iāll talk with Jakub and come back to you on this. Just note Iāll be on vacation next week (18.09 - 22.09)
Regards, PaweÅ
Thanks @pgesek!
One other thing that occurred to meā¦ have there been thoughts on how to handle conflicts? Apologies if I missed the details in the documentationā¦
Currently Sync 1.0 doesnāt really have a strategy for resolving conflicts (beyond ālast in winsā)ā¦ it would be great if we could at least flag conflicts (perhaps based on dateCreated and dateChanged?). Conflict resolution can certainly become complex, so Iād understand keeping it simple for Phase 1, but if we could have something better than the current ālast in winsā (without any notification) that would be great.
Take care, Mark
(More comments, which I wrote over several days, so are probably chaotic)
āMetadata resource listā comments:
āSync 2.0 Architecture Overviewā comments:
synchronization can be done in two directions ā¦ support both independently, as one-way sync is being used in the field currently by implementations adopting Sync 1.0. ā¦ All synchronization will be initiated by slaves, independent from the direction that data is transmitted
āSync 2.0 configuration variablesā comments:
sync.enabled that controls this.About error handling: are we going to preserve the downloaded REST/FHIR response to replay again? Or would we re-fetch it from its url?
The event module uses ActiveMQ, which has been occasionally buggy for us. (Maybe weāre just using a 5 year old version of it.) We should definitely consider either modernizing the event module, or writing a from-scratch replacement, that can be used for Sync 2.0 and other things too. But for Sync 2.0 the real requirements is to have something that goes from hibernate interceptor events to atom feed events. We could go via a message broker like in the event module, which could be more reusable, but also adds (unnecessary?) complexity.
Good point. So, yes, itās fair to prioritize logging. I wouldnāt over-complicate by giving too much configuration about which loggers to use. Just pick a good default, at least in the first pass.
I was also thinking about this. Personally I would prioritize āquicker to developā above āconflict resolutionā. (Ultimately, it would be nice to capture a complete change history in openmrs-core, or in a multi-purpose module, not just for sync.)
Even to do something simple like comparing based on dateCreated/dateChanged requires that we persist an extra piece of data on each update (i.e. what was its original created/changed timestamp) so we can do a 3-way comparison when merging. If this can be done while generating the atom feed, without adding lots of complexity, it could be worthwhile.
I wonder if we can leave a hook to add conflict resolution in at a later date, once a separate module has been built that captures a full change log.
Agree that āconflict resolutionā could be a bit of a beast and we shouldnāt get too bogged down with doing it perfectly. However I think we certainly should consider have at least some sort of conflict notification in Phase 1 or at least some sort of explicit warning to prevent people from doing ādangerousā things.
Sync 1.0 has no conflict resolution or warnings, and the general rule of thumb with Sync 1.0 is that āyou never should be editing the same patient on both the parent and the childā (or two children, for that matter)ā¦ but I find myself forgetting that rule a lot, no mind an everyday end user. Itās kind of scary because it can silently lead to data corruption (and I have no way of saying whether that has actually happened in our current implementations).
Take care, Mark
A few quick thoughts:
If we canāt use the event module for thisā¦what exactly is the purpose of the event module? If we think the implementation of the event module is buggy, then we should fix the implementation, right? It would be nice for us to deal with hibernate interceptor messiness once, and do it properly, in the event module, and then for us to rely on that for a way to publish / subscribe to changes.
Presumably, there could be an Event listener that we support which would simply hit the REST endpoint, get the JSON, and log this - maybe to an external document database, along with the other information from the event (event type, data type, uuid, etc). Then, implementations that want a full audit log and have the terabytes to spare could simply enable this. This would seem like it would be pretty low-effort to add in.
Mike