Testing MergePatientData Module

[samuel34 (Samuel Male)]

You mean I modify the default value to null?

[samuel34 (Samuel Male)]

In the sql file, here is the sql script that creates the person_address table.

DROP TABLE IF EXISTS `person_address`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `person_address` (
`person_address_id` int(11) NOT NULL AUTO_INCREMENT,
`person_id` int(11) DEFAULT NULL,
`preferred` tinyint(1) NOT NULL DEFAULT '0',
`address1` varchar(255) DEFAULT NULL,
`address2` varchar(255) DEFAULT NULL,
`city_village` varchar(255) DEFAULT NULL,
`state_province` varchar(255) DEFAULT NULL,
`postal_code` varchar(50) DEFAULT NULL,
`country` varchar(50) DEFAULT NULL,
`latitude` varchar(50) DEFAULT NULL,
`longitude` varchar(50) DEFAULT NULL,
`creator` int(11) NOT NULL DEFAULT '0',
`date_created` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
// And more other columns.

I have checked almost all tables have that default value for ‘date_created’ column.

Cause of Error

Running this .

mysql> SHOW VARIABLES LIKE 'sql_mode' ;

Produces such.

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

That causes the error by rejecting that default date. So what were you saying I should do @dkayiwa?

[samuel34 (Samuel Male)]

This issue has been resolved @dkayiwa

Running this solved the issue.

SET GLOBAL sql_mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION";

[samuel34 (Samuel Male)]

On invoking an Export operation on a server with over 2000+ Patients, 146243 Obs etc, I hit upon this exception

java.lang.OutOfMemoryError: Java heap space

I tried looking for memory leaks in my codebase but I think I found non. I then resorted to increasing Java Options like

export JAVA_OPTS="-Xms6144m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=356m - XX:PermSize=256m -XX:MaxPermSize=356m"

This didn’t work for me, I then tried it with Maven like:-

export MAVEN_OPTS="-Xms6144m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"

How ever when I restart my server, it looks like maven doesn’t use these options.

Using JAVA_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre

Using MAVEN_OPTS:   -Xmx768m -XX:MaxPermSize=512m

Could I be missing something @dkayiwa & @ssmusoke ?

[samuel34 (Samuel Male)]

I set the variable successfully but this isn’t as efficient as I expected.

I’m optimizing my code

[ssmusoke (Stephen Senkomago Musoke)]

@samuel34

I set the variable successfully but this isn’t as efficient as I expected.

1. Which variable did you set

2. How did you set it manually?

3. To what value did you set it?

[samuel34 (Samuel Male)]

Hi,

Thanks @ssmusoke for the dump file you shared. This actually has given me the real picture of how the module behaves. Actually there is something am missing now.

I don’t know what I should handle this but when exporting data, the Concept Resource makes the resulting data so bulky (Around 1GB) on a server with 2000+ Patients but without Concepts, its around 80MBs. Remember we have a limit of the file size which 75MB. I had not realized this before testing with large data.

So how should I handle this?

cc: @dkayiwa

[samuel34 (Samuel Male)]

@ssmusoke

Which variable did you set

MAVEN_OPTS

@ssmusoke

To what value did you set it?

Xmx4g

@ssmusoke

How did you set it manually?

I just put it in my ‘.bash_profile’ file

[samuel34 (Samuel Male)]

The challenge is we are limited by the file size.

Is there anyway we could Identify Concepts in the Concept Dictionary(the Concepts that are likely to be the same from one server to another) so that I just transfer Concept Objects with Identifiers. What I have realized is that almost all the Concepts are already present in all refApp distros.

I don’t know whether I make any sense by that!

cc: @ssmusoke, @dkayiwa

[samuel34 (Samuel Male)]

I’m still curious of how I’m gonna handle this @dkayiwa. Looking back at project wiki, I hit upon this.

Site level users and metadata are not synced, neither are concepts and forms, as they are expected to be similar.

I’m more interested in the Concept part of it. Could you shade more light here? What of those customized Concepts?

[samuel34 (Samuel Male)]

Here are my thoughts.

Since we are not using REST injections, we should export all the Concept data. But while importing, we first make a sanity check to see whether it already exists before we Merge a Concept otherwise I don’t see away to distinguish between Customized Concepts and General Concepts. But Exporting Concepts makes the encrypted file extremely large so we should fix

Overriding the default multipart file size

Hi, With the mergepatientdata module, you can export a good number of patient and encounter, I dowloaded over 100 patients with their encounter and these made up a file of around 89mbs, on trying to upload this file file to another server, I hit upon this org.springframework.web.multipart.MaxUploadSizeExceededException: Maximum upload size of 75000000 bytes exceeded; nested exception is org.apache.commons.fileupload.FileUploadBase$SizeLimitExceededException I thought by default the spring mu…

cc: @ssmusoke, @dkayiwa

[dkayiwa (Daniel Kayiwa)]

Are you trying to do what this module already does? https://wiki.openmrs.org/display/docs/Metadata+Sharing+Module

[samuel34 (Samuel Male)]

Ooopss, seems thats the mistake I have been doing. All along I have been wasting a lot of time and Resources reinventing the wheel

I wish I knew this before!

@ssmusoke also went ahead and explained to me this today.

What next

This actually has made my work simple. All I’m to do is to assume that metadata like Concept is already present in all the servers(Could be by the metadata sharing module). That being true, I should just go ahead and Map Concepts, Users, Forms… etc using Identifiers.

Hope this makes sense @dkayiwa