Dockerised demo refapp, and other questions about demo data

Due to some problems with demo (Process to update the demo server), I decided to spend my weekend trying to dockerise refapp in such a way we could replace the current deployment to use docker (and replace it completely daily or so).

The refapp standalone doesn’t particularly help, because it’s graphical, requires input from user, provides database as files in filesystem (not a sql), and expects database on the localhost (and I’d rather follow docker philosophy of one process per container).

This is what I got so far: https://github.com/cintiadr/docker-openmrs

Firstly, I discover that mysql 5.7 fails to generate database, probably it’s a known issue.

Questions:

  1. Is there a way to convince openmrs to create tables and demo data without showing the wizard?
  2. Do I need to generate a new dump on every new version of reference application or is it safe to use an older dump, and allow openmrs to apply the delta on top of it?
  3. Would it be acceptable to generate a .sql inside the standalone? Or is there any recommended way to generate the empty database and demo data automatically?
  4. What exactly do I need to do to use https://wiki.openmrs.org/display/RES/Demo+Data data? I cannot use it on refapp apparently, and also if I try to use on platform it fails on missing classes. How that’s generated, and how to use it?

Thanks again @cintiadr for spending your weekend on this! :slight_smile:

Below are the responses to your questions:

The MySQL 5.7 issue was fixed in the openmrs platform 2.0.1 release. So it should all be well with the latest stable releases. https://issues.openmrs.org/browse/TRUNK-4813

  1. There is a provision for an installation script using a system property named “OPENMRS_INSTALLATION_SCRIPT”, as you can see from here and below: https://github.com/openmrs/openmrs-core/blob/master/web/src/main/java/org/openmrs/web/filter/initialization/InitializationFilter.java#L318

  2. As far as i know, one should be able to upgrade their reference application data with a newer version of the reference application, without having to generate a new dump.

  3. You should be able to run the standalone in a non interactive mode. Take a look at the command line mode options at https://wiki.openmrs.org/display/docs/OpenMRS+Standalone

  4. This demo data is only for the platform, not reference application. I recently updated it with the demo data for the latest platform 2.0.1 release. If you get any problems with it, i would love to look at the error log.

  1. The SDK does this at startup so maybe you can try to create a server with that which I think is better than the standalone process.

  2. Not a docker expert, but you could use the SDK to update the versions deployed to a docker container to recreate it maybe @raff can help here.

You should be able to reproduce using branch mysql-7. It downloads the platform, all modules, and start openmrs refapp with an empty database. You need to go to the browser, run the wizard manually (hostname: database, username: openmrs, password: 0p3nmrs) to generate tables and demo data.

Here the full log: https://gist.github.com/cintiadr/092dd4403ea0f081f2975374899dab68 Looks related to https://www.percona.com/blog/2016/10/18/upgrading-to-mysql-5-7-beware-of-the-new-strict-mode/

Thanks! I will take a look.

The standalone is not intended for production use, right? That’s what the download page says. While I’d love to have an standalone which does have tomcat, platform, modules and jre and deploy that to prod, it’s very suspicious to have a non-security-patched database, installed in a non-standard way, with dump provided in a non-standard way too. I can try it our regardless, it could be a good test to have in our CI.

https://gist.github.com/cintiadr/fdbe99b746b84dd1a5f531e7e4527109 for the logs. It downloaded platform 2.0.1, imported demo data and started the container. Nothing fancy. You should be able to reproduce using branch platform-only-demo.

In the past, I’ve tried to create docker images using the maven SDK. It wasn’t pretty nice. In order to have a build, I needed maven, and then maven would spend a lot of time downloading the internet. I could of course mount ~/.m2, but then I would have another folder to take care, and to share if I needed faster builds, plus caring about m2 repo poisoning, cleaning up snapshots, and so on. Adding to the fact that would be pretty far away of what any production-like system. One doesn’t install maven in production. :smiley:

Unless the SDK itself would happily provide a sql dump of the demo data, which could be generated from CI and deployed somewhere.

@cintiadr, here is how you would achieve all that with SDK:

mvn openmrs-sdk:build-distro -Ddistro=referenceapplication:2.5 -Ddir=demoserver -DdbSql=demo-data.sql
cd demoserver
docker-compose up

That’s it! More info in SDK docs.

If you need to build a distro then you should be using maven and SDK. When you deploy distro in production you just use docker to fire it up.

I’d love us to use something like https://www.docker.com/products/docker-cloud to manage containers running on our machines. I’m in the process of configuring devtest04 to evaluate that.

@cintiadr the demo dataset is not for the reference application. If you want demo data for the reference application, set the global property named “referencedemodata.createDemoPatientsOnNextStartup” to the number of demo patients that you want. This should create some observations along with the patients.

As for MySQL, which exact version are you using? Is it equal to or greater than 5.7.16?

If I knew this before, it would have saved me so much time! I got to the point of almost exactly the same thing that the SDK generates :smiley: So, what exactly is the plan with the generated docker compose? Would it be that people would commit that (and get it build by dockerhub or a CI)? Use it as initial Docker files to customise per distro? Can I do the same with, let’s say, with https://github.com/openmrs/openmrs-distro-platform/ ?

Also, it doesn’t have the demo data straightaway, it can generate the data.

[Edit: I’m using docker datacenter, not cloud, so ignore what I said]

While https://blog.docker.com/2016/11/docker-aws-public-beta/ could be something useful, I’d rather just have ECS (https://aws.amazon.com/ecs/) if we don’t need service discovery or something.

Maybe it’s not worth yet. I digress.

I know that, that’s why I just imported that data when only the platform was deployed, as you can see on the
platform-only-demo branch. Platform only here, and exceptions on the logs I sent with missing classes.

By https://hub.docker.com/_/mysql/, it’s 5.7.16 exactly.

SDK uses openmrs-distro.properties to generate docker files and fetch relevant modules/war, e.g. https://github.com/openmrs/openmrs-distro-referenceapplication/blob/master/package/src/main/resources/openmrs-distro.properties

Distros can modify such a file to include/update custom modules and then recreate docker files, which can be distributed in any way (preferably as built images in dockerhub).

The demo data needs to be generated the way @dkayiwa described with the referencedemodata module and afterwards you should do a dump and store it in a repo. You do that manually and just once.

https://github.com/openmrs/openmrs-distro-platform/ does not yet contain openmrs-distro.properties, but it should be migrated to use that.

This is what I did for that https://github.com/openmrs/openmrs-sdk/pull/119

Changing the SDK to allow deployment of docker images to dockerhub, with more flexibility on the docker compose file.

The SQL generation still not a solved problem.

Hi @raff,

I tried to run the above set of commands on our own custom distro and on a recent version of our production database, like this:

mvn openmrs-sdk:build-distro -Ddistro=/path/to/openmrs-distro.properties -Ddir=demoserver -DdbSql=/path/to/openmrs.sql

Then

cd demoserver && docker-compose up

But then it fails and gets stuck on the maintenance page:

Here is the log (that loops on these errors): SDK & docker: java.net.ConnectException: Connection refused (Connection refused) · GitHub Could you help me make that work?

Also, for when it works ;-), how can I change the Tomcat port from 8080 to anything else?

Anybody from the @SolDevelo team that could assist me with launching our custom distro with SDK-docker?

The issue might be that mysql didn’t start yet. Could you please try editing startup.sh by changing /usr/local/tomcat/wait-for-it.sh --timeout=120 mysql:3306 to /usr/local/tomcat/wait-for-it.sh --timeout=180 --strict mysql:3306?

Let us know if it fixes the issue for you so we can make the change in SDK.

Please create an issue in SDK for changing the default Tomcat port. For now you can edit docker-compose.yml by changing:

ports: 
      - "8080:8080" 

to

ports: 
      - "8090:8080" 

It will expose Tomcat’s 8080 port under 8090 port on your host machine.

Ok, thanks, just tried that. First of all I think that the timeout change to startup.sh didn’t apply as I’m seeing this anyway:

openmrs-lfhcdistro          | wait-for-it.sh: waiting 120 seconds for mysql:3306
openmrs-lfhcdistro          | wait-for-it.sh: mysql:3306 is available after 0 seconds
openmrs-lfhcdistro-mysql    | 2016-12-16 10:41:54 0 [Note] mysqld (mysqld 5.6.35) starting as process 1 ...
openmrs-lfhcdistro-mysql    | 2016-12-16 10:41:54 1 [Note] Plugin 'FEDERATED' is disabled.

However now it seems to work… but then I’m getting this later on: https://gist.github.com/mks-d/61616cabcc76c2afaf5e09fdc78f53f0

Forgot to mention you need to run docker-compose build to rebuild the image after modifying scripts.

The error you get now indicates something is wrong with the SQL you imported… How did you generate that? For some reason person_attribute_type table is missing.

To be more precise the SQL you import should be generated from a valid OpenMRS database. We then run liquibase-update-to-latest.xml on it to make sure it is up to date with the version of openmrs-api you are running.

EDIT: Actually, it is possible that your SQL is correct, but the first time you ran docker-compose up it wasn’t properly imported due to mysql being unavailable. Please drop containers and try again.

docker-compose kill
docker-compose rm -f
docker-compose build
docker-compose up

Ok yes I was about to confirm that the table indeed existed in our database. Ok I will start again the whole process and let you know how it goes.

@raff, I followed your steps (actually I also even started from scratch again) and I still get the same error. The full log is here:

Is it normal that there is this prefix ‘openmrs.’ in “Table 'openmrs.person_attribute_type' doesn't exist”?

‘openmrs.’ is a schema name so it’s ok. Is your distro file and initial sql public so I can try reproducing the issue?

@raff, could I possibly follow up with you IRC about this? I guess it’d be easier.

I can report two things at this stage.

The first time ever I run docker-compose up (with a timeout set last time at 1,800) it always eventually fails saying that root can’t connect to the database. So from there I do:

docker-compose stop
docker-compose kill # not sure if still needed after stop
docker-compose rm -f
docker-compose build
docker-compose up

Then it all goes very quick to the Tomcat container and I am prompted to load the webapp. When doing so the ‘update database’ fails because a table or another is missing. And that’s clearly because 30 min were not enough! With 30 min it goes up to filling the obs table and ends up being interrupted in the middle of that one.

Clearly I can set a timeout even higher (I guess 1 hour would work), but why does it take so long for the MySQL container to import a SQL dump file?? If I do this outside of Docker it takes like 3/4min.

Any chance you can get to the mysql configuration? You may need to increase the innodb_buffer_pool_size to a larger value 40% RAM to speed loading of the dump especially if it’s single inserts