Testing MergePatientData Module

Which version of MySQL are you running?

Server version: 5.7.17 MySQL Community Server (GPL)

Can you paste the entire log at pastebin.com?

@dkayiwa here we go https://pastebin.com/0i87Z3D3

When you look at your person_address table, what is the default value for date_created?

Ooops seems this is the culprit. The deafualt value is ‘0000-00-00 00:00:00’ @dkayiwa

1 Like

Can you remove it and try again?

You mean I modify the default value to null?

In the sql file, here is the sql script that creates the person_address table.

DROP TABLE IF EXISTS `person_address`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `person_address` (
`person_address_id` int(11) NOT NULL AUTO_INCREMENT,
`person_id` int(11) DEFAULT NULL,
`preferred` tinyint(1) NOT NULL DEFAULT '0',
`address1` varchar(255) DEFAULT NULL,
`address2` varchar(255) DEFAULT NULL,
`city_village` varchar(255) DEFAULT NULL,
`state_province` varchar(255) DEFAULT NULL,
`postal_code` varchar(50) DEFAULT NULL,
`country` varchar(50) DEFAULT NULL,
`latitude` varchar(50) DEFAULT NULL,
`longitude` varchar(50) DEFAULT NULL,
`creator` int(11) NOT NULL DEFAULT '0',
`date_created` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
// And more other columns.

I have checked almost all tables have that default value for ‘date_created’ column.

Cause of Error

Running this .

mysql> SHOW VARIABLES LIKE 'sql_mode' ;

Produces such.

ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

That causes the error by rejecting that default date. So what were you saying I should do @dkayiwa?

This issue has been resolved @dkayiwa

Running this solved the issue.

SET GLOBAL sql_mode="ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION";
1 Like

On invoking an Export operation on a server with over 2000+ Patients, 146243 Obs etc, I hit upon this exception

java.lang.OutOfMemoryError: Java heap space

I tried looking for memory leaks in my codebase but I think I found non. I then resorted to increasing Java Options like

export JAVA_OPTS="-Xms6144m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=356m - XX:PermSize=256m -XX:MaxPermSize=356m"

This didn’t work for me, I then tried it with Maven like:-

export MAVEN_OPTS="-Xms6144m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"

How ever when I restart my server, it looks like maven doesn’t use these options.

Using JAVA_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre

Using MAVEN_OPTS:   -Xmx768m -XX:MaxPermSize=512m

Could I be missing something @dkayiwa & @ssmusoke ?

I set the variable successfully but this isn’t as efficient as I expected.

I’m optimizing my code

  1. Which variable did you set

  2. How did you set it manually?

  3. To what value did you set it?

Hi,

Thanks @ssmusoke for the dump file you shared. This actually has given me the real picture of how the module behaves. Actually there is something am missing now.

I don’t know what I should handle this but when exporting data, the Concept Resource makes the resulting data so bulky (Around 1GB) on a server with 2000+ Patients but without Concepts, its around 80MBs. Remember we have a limit of the file size which 75MB. I had not realized this before testing with large data.

So how should I handle this?

cc: @dkayiwa

MAVEN_OPTS

Xmx4g

I just put it in my ‘.bash_profile’ file

The challenge is we are limited by the file size.

Is there anyway we could Identify Concepts in the Concept Dictionary(the Concepts that are likely to be the same from one server to another) so that I just transfer Concept Objects with Identifiers. What I have realized is that almost all the Concepts are already present in all refApp distros.

I don’t know whether I make any sense by that!

cc: @ssmusoke, @dkayiwa

I’m still curious of how I’m gonna handle this @dkayiwa. Looking back at project wiki, I hit upon this.

Site level users and metadata are not synced, neither are concepts and forms, as they are expected to be similar.

I’m more interested in the Concept part of it. Could you shade more light here? What of those customized Concepts?

Here are my thoughts.

Since we are not using REST injections, we should export all the Concept data. But while importing, we first make a sanity check to see whether it already exists before we Merge a Concept otherwise I don’t see away to distinguish between Customized Concepts and General Concepts. But Exporting Concepts makes the encrypted file extremely large so we should fix

cc: @ssmusoke, @dkayiwa

Are you trying to do what this module already does? https://wiki.openmrs.org/display/docs/Metadata+Sharing+Module

Ooopss, seems thats the mistake I have been doing. All along I have been wasting a lot of time and Resources reinventing the wheel :rage:

I wish I knew this before!

@ssmusoke also went ahead and explained to me this today.

What next

This actually has made my work simple. All I’m to do is to assume that metadata like Concept is already present in all the servers(Could be by the metadata sharing module). That being true, I should just go ahead and Map Concepts, Users, Forms… etc using Identifiers.

Hope this makes sense @dkayiwa