Bahmni Connect: new offline sync strategy

mksrom · January 10, 2018, 3:04pm

Hi Bahmni devs!

I am looking at creating a new offline Sync Strategy that would simply allow to sync all patients, like a ‘AllPatientsSyncStrategy’ or something.

It seems like I need to override the 3 following methods:

getFilterForDevice
getEventCategoriesList
getEventLogsFromEventRecords

Anyone could help me identify how these functions work and what they can return?

(tagging the main commiters @shruthipitta, @sumanmaity112)

sumanmaity112 · January 10, 2018, 4:32pm

Hi @mksrom,

getFilterForDevice determines all the filters for a login location for all types of events. In case of LocationBasedSyncStrategy, it returns all the address code. For example, if you log in to a 3rd level location it will give all the childs address code (4th level, 5th level etc).

getEventCategoriesList returns which types of events you want to sync to your offline devices. Your database may have several types of events (patient, encounter, addressHierarchy, forms), but you are interested to particular some types of events (like- patient, encounter, offline-concepts etc). You will set those events in this function. Example can be found here.

getEventLogsFromEventRecords, this method does really simple job. It just read the event_records table and copy those events to event_log table with proper filter.

For your case, getFilterForDevice can return a empty list and getEventLogsFromEventRecords should copy all the patient and encounter events can be copy to event_log table wiith null as filter.

mksrom · January 11, 2018, 12:43pm

Thanks for the explanation.

Very often in Bahmni Connect app code, there is the mention of ‘markers’ or ‘marker’.

Such as here:

github.com

Bahmni/bahmni-connect/blob/2cd8e6800f9d7b3bff480ec7a6b0f2e1bbbaac68/ui/app/common/offline/services/offlineSyncService.js#L74


};


var getRemainingFileNames = function (fileNames, synced) {
    var remaining = _.difference(fileNames, synced);
    return remaining.length ? remaining : fileNames.length ? [_.last(fileNames)] : fileNames;
};


var savePatientDataFromFile = function () {
    var eventLogUuid;
    var defer = $q.defer();
    offlineDbService.getMarker('patient').then(function (marker) {
        if (marker.lastReadEventUuid) {
            return defer.resolve();
        }


        return getDbName().then(function (dbName) {
            var promises = marker.filters.map(function (filter) {
                var syncedInfo = offlineService.getItem("synced") || {};
                var synced = syncedInfo[dbName] || [];
                return $http.get(Bahmni.Common.Constants.preprocessedPatientFilesUrl + filter).then(function (response) {
                    return getPatientDataForFiles(getRemainingFileNames(response.data, synced), 0, null, dbName).then(function (uuid) {

What are markers?

sumanmaity112 · January 11, 2018, 1:17pm

Hi @mksrom,

**markers ** are like pointers, which holds the lastReadEventUuid for a event_type. It helps the device to determine the position of synced events. In an offline device, there is a table called event_log_marker which holds the markers for all types of events. Check the following dummy event_log_marker table -

Here I have 3 types of event - "patient", "addressHierarchy", "encounter". During sync device will send marker name and lastReadEventUuid, the backend server will give all the pending events after lastReadEventUuid if any.

mksrom · January 11, 2018, 1:47pm

Great thanks. Makes things clearer and clearer.

I have just created the AllPatientsSyncStrategy and I am having issues:

The offlineDbService.getMarker('patient') returns an empty “marker” when evaluated.

Indeed my event_log_marker table is empty at this point:

When/where is the event_log_marker table filled, and under which conditions?

sumanmaity112 · January 11, 2018, 2:42pm

Hi @mksrom,

There are two databases metadata and Bahmni. metadata is used to hold common types of events, like - concepts and forms. Bahmni database used to hold other events like - patient, encounter etc.

Can you check the Bahmni database? Also, check is there any event in event_log table in server side with that filter.

mksrom · January 11, 2018, 2:54pm

Thanks @sumanmaity112.

The Bahmni database has all its tables empty except for the patient_attribute_type (which has 10 entries).

Where does the initialization of the DB first happens?

sumanmaity112 · January 11, 2018, 4:13pm

Hi @mksrom,

Bahmni table will be updated once initial sync is completed.

Google Chrome console has some refresh issue, please click on refresh icon (↻) or close and reopen the developer console

Please check is there any event in event_log table of server side. If there is no event please follow this link.
There should be some call about category, url should be <host name>/openmrs/ws/rest/v1/eventlog/filter/category. Check the output. The output will be something like ["patient","encounter","addressHierarchy","offline-concepts","forms"]
There will be another call to get the filter, url will be <host name>/openmrs/ws/rest/v1/eventlog/filter/markers/<provider uuid>/<addressUuid>/<loginLocationUuid>. Check the output. The output will be looks like {"patient":["GAN"],"offline-concepts":[],"addressHierarchy":[],"encounter":["GAN"],"forms":[]}

Once step 2, 3 is done, you can check local storage. It will be looks like -

Now check the event_log table is there any event with proper filter (filter can be found in step 3).

All the sync and database related stuff happens in offlineDbService and offlineSyncService.

mksrom · January 11, 2018, 5:50pm

(thanks @sumanmaity112 for taking the time to explain me all this)

[root]# curl http://localhost/event-log-service/
Event log service is up and running.

So that part is fine.

The content of the ‘event_log’ table in the server database seems valid, here is the last entries:

| 1604 | 2018-01-10 22:38:53 | /openmrs/ws/rest/v1/reference-data/drug/53ab7e2c-a58b-4db7-951f-c7c7328c6299                        | drug      | NULL   |
| 1605 | 2018-01-10 22:38:53 | /openmrs/ws/rest/v1/reference-data/drug/77182d37-35a6-41da-9687-382a3f544f84                        | drug      | NULL   |
| 1606 | 2018-01-10 22:38:53 | /openmrs/ws/rest/v1/reference-data/drug/8970848f-926f-410a-ad77-c8eb499bf3a2                        | drug      | NULL   |
| 1607 | 2018-01-10 22:38:53 | /openmrs/ws/rest/v1/reference-data/drug/e302c34b-ff13-468d-aa50-dbde6736e0b5                        | drug      | NULL   |
| 1608 | 2018-01-10 22:38:54 | /openmrs/ws/rest/v1/reference-data/drug/56fdeba8-c28f-40a8-a977-6c78dbfd92a4                        | drug      | NULL   |
| 1609 | 2018-01-10 22:38:54 | /openmrs/ws/rest/v1/reference-data/drug/fd060b29-c849-44bf-87c4-e8c2dda2ca4b                        | drug      | NULL   |
| 1610 | 2018-01-10 22:38:54 | /openmrs/ws/rest/v1/reference-data/drug/d51a819c-0800-435e-bc80-65c8431ac236                        | drug      | NULL   |
| 1611 | 2018-01-11 20:00:14 | /openmrs/ws/rest/v1/patient/1c788c8a-79c5-4ed7-a76e-c0b817152ae8?v=full                             | patient   | NULL   |
| 1612 | 2018-01-11 20:39:02 | /openmrs/ws/rest/v1/patient/1c788c8a-79c5-4ed7-a76e-c0b817152ae8?v=full                             | patient   | NULL   |
| 1613 | 2018-01-11 20:39:08 | /openmrs/ws/rest/v1/bahmnicore/bahmniencounter/119bafb4-7a08-407f-b9e3-c3e6abea6c3e?includeAll=true | Encounter | NULL   |
+------+---------------------+-----------------------------------------------------------------------------------------------------+-----------+--------+
1613 rows in set (0.00 sec)

Looks like all is good here too.

There is one call indeed:

https://localhost/openmrs/ws/rest/v1/eventlog/filter/category

Response:

["patient","encounter","addressHierarchy","offline-concepts","forms"]

One call made with 200 status code:

https://localhost/openmrs/ws/rest/v1/eventlog/filter/markers/c1c26908-3f10-11e4-adec-0800271c1b75/010c1dbe-238b-41e0-9229-f1a26682d1cd/1d0d52ce-f171-4be4-b91a-75c537238e89

But no response at all. Not an empty arrray, but simply nothing. Looking at it…

[EDIT]

This response is what comes from getFilterForDevice() method of my AllPatientsSyncStrategy class. It indeed returns ‘null’. Trying with returning an empty object rather than null.

    public Map<String, List<String>> getFilterForDevice(String providerUuid, String addressUuid, String loginLocationUuid) {
    	Map<String, List<String>> categoryFilterMap = new HashMap<String, List<String>>();
        categoryFilterMap.put("patient", new ArrayList<String>());
        categoryFilterMap.put("encounter", new ArrayList<String>());
        categoryFilterMap.put("addressHierarchy", new ArrayList<String>());
        categoryFilterMap.put("offline-concepts", new ArrayList<String>());
        categoryFilterMap.put("forms", new ArrayList<>());
        return categoryFilterMap;
    }

Response:

{"patient":[],"offline-concepts":[],"addressHierarchy":[],"encounter":[],"forms":[]}

That does solve my problem!

I come across another error though that I will investigate and detail tomorrow if needed.

mksrom · January 11, 2018, 6:12pm

The new error I encounter is the following:

Error 400 - Bad Request
https://localhost/event-log-service/rest/eventlog/events/encounter

Response message:

{"timestamp":1515694082657,"status":400,"error":"Bad Request","exception":"org.springframework.web.bind.MissingServletRequestParameterException","message":"Required String[] parameter 'filterBy' is not present"}

“Required String parameter ‘filterBy’ is not present”

Indeed the resource in the event-log-service module does require “filterBy” parameter:

github.com

Bahmni/event-log-service/blob/75c189832b60d88a2f461f840b456152d1f63a8a/event-log-service-webapp/src/main/java/org/bahmni/module/eventlogservice/controller/EventLogController.java#L21


      
          @RequestMapping("/rest/eventlog")
          public class EventLogController {
              private EventLogRepository eventLogRepository;
          
              @Autowired
              public EventLogController(EventLogRepository eventLogRepository) {
                  this.eventLogRepository = eventLogRepository;
              }
          
              @RequestMapping(value = "/events/{category}", method = RequestMethod.GET)
              public Map<String, Object> getEvents(@PathVariable String category, @RequestParam(value = "uuid", required = false) String uuid, @RequestParam(value = "filterBy", required = true) String[] filterBy) {
                  List<String> categoryList = new ArrayList<String>();
                  categoryList.add("addressHierarchy");
                  categoryList.add("patient".equals(category) ? "Encounter" : "patient");
                  return getEventsExcept(uuid, filterBy, categoryList);
              }
          
              private Map<String, Object> getEventsExcept(@RequestParam(value = "uuid", required = false) String uuid, @RequestParam(value = "filterBy", required = true) String[] filterBy, List<String> categoryList) {
                  Map<String, Object> response = new HashMap();
                  if (uuid == null) {
                      response.put("events", eventLogRepository.findTop100ByFilterInAndCategoryNotIn(Arrays.asList(filterBy), categoryList));

However, since my filter is supposed to return every encounter (and every patient), I think it’s normal to have ‘filterBy’ to be empty.

What do you think about this @sumanmaity112?

sumanmaity112 · January 12, 2018, 4:11am

Hi @mksrom,

The logic behind filterBy is a required param as the server will have lots of events. One device will not be able to hold of these events and also you will not use the same device for all the login location. There you need to filter the events depends on login location.

Can you pass filterBy param as an empty array ([])?

I mean can you try this <host name>/event-log-service/rest/eventlog/events/patient?filterBy=&uuid=<some patient event uuid>

mksrom · January 12, 2018, 11:17am

When trying the request with filterBy=, it can go through.

[root]# curl -k https://localhost/event-log-service/rest/eventlog/events/encounter?filterBy=

{"pendingEventsCount":0,"events":[]}

However, it returns empty results while I would expect it to return all encounters. Not sure I make correct use of the filterBy here.

(I have created a patient already and as we can see in the event_log database table pasted in a previous post, there is one “Encounter” event)

As I am playing around with the event-log-service module, I am able to easily try different values of filterBy in the Integration Tests (here). I confirm again that an empty filterBy returns empty response.

What are the possible filterBy values that I can use?

I my use case, our clinic is quite small and it is a mobile clinic, we cannot have as many login location as there is villages they visit, so we have chosen to have only one generic location named ‘Health Center’ (See this post where I explain and investigate the LocationBasedSyncStrategy and end up thinking this strategy does not fit for us).

Ideally, I could have something where only the “less than 1 year old” encounters would be fetched for instance. But in the mean time I am ok with having all encounters events fetched.

mksrom · January 12, 2018, 4:45pm

From a Skype call with @sumanmaity112 I will answer to myself here (let me know if I make mistakes)

There is no “possible values of filterBy” because the filter value of each event is provided by the sync strategy itself.

Anytime a new event happens in OpenMRS it is broadcast from the OpenMRS scheduler. The event is evaluated by the sync strategy method getEventLogsFromEventRecords(). So depending of what the strategy rules are, the filter is set. (The value of the filter is arbitrary – see below).

The filter is a key that will be used to retrieve specific events on a device that matches this key (I believe by exact match).

So both events and devices have filters.

The events to be synced on a specific device are the events that match the device filter.

To get the current device filter, the sync strategy implements getFilterForDevice() method. For instance, in the case of ID based sync strategy, the device filter is the login location ID prefix (“GAN”, “BAM”, etc…)

Based on this, in order for me to sync all patients and encounters, I have to set an arbitrary filter string for every event of every type. This happens in the getEventLogsFromEventRecords() method.

Then I have to configure the getFilterForDevice() method to always return the same filter string value.

Doing so will make every event available on any device.

sumanmaity112 · January 12, 2018, 4:58pm

@mksrom,

You are right. We do the exact match for filter.

mksrom · January 13, 2018, 3:14pm

One more question…

When is event_log table contents created the first time?

At the creation of new server? At the restart? After running a specifc command?

sumanmaity112 · January 13, 2018, 3:51pm

Hi @mksrom,

When you install bahmni-event-log-service, its automatically creates event_log service through migration. Check this link

shruthipitta · January 15, 2018, 11:39am

@mksrom if you when is event log table populated with data => event log service poll event_records for new events and it copies event from there to event_log and associates a filter to it.

mksrom · January 15, 2018, 2:37pm

Thank you. Yes that’s what I meant.

I am just asking because I want to know if this table is “re-generated” sometimes and each event is reprocessed at some point.

That’s because ultimately I would like to sync only “recent” encounters (less than one year old). And therefore I would like each event to be re-evaluated at some point so older encounters don’t have the matching filter.

Do you think that’s possible?

sumanmaity112 · January 15, 2018, 5:26pm

Hi @mksrom,

If you edit anything on UI (like patient data, observations) it will try to give re-evaluate the filter.

If you want to re-evaluate filters for all the events, you can truncate your event_log table and restart the bahmni-event-log-service.