hello @wolf been running this command but getting out of memory error that am still trying to fix , may be @dkayiwa or some one can help build and provide you the logs of the full build as i also crack down this error and get the JVM to display the heap size
Hey there, I remembered how to get around the mxj database issue and rewired the build to use the local MySQL database.
The Liquibase library is outdated, openmrs-core uses version 3.10.2 now.
mvn clean package -Dopenmrs.version=2.4.0-SNAPSHOT fails when using 3.10.2 as the Liquibase API has changed since 2.0.1.
I look into that over the weekend.
thanks @wolf for the feedback on this 
sounds positive
@dkayiwa and @gcliff I spent some time with the failing build for openmrs-standalone and would like to share my findings with you.
a) openmrs-core and openmrs-standalone use different Liquibase versions
openmrs-core uses Liquibase 3.10.2 whereas openmrs-standalone still uses 2.0.1. This results in a failing build as there were a few breaking changes between Liquibase 3.x. amd 2.x.
The fix is to upgrade the Liquibase version used by openmrs-standalone.
b) openmrs-standalone still uses the 1.9.x snapshot change logs and all update-to-latest change logs starting from 2.0.x.
Change sets in liquibase-update-to-latest-2.0.x.xml collides with the DDL contained in openmrs_concepts_1.9.9_20171118.sql.
The CIEL DDL defines a non-null constraint on concept_datatype.uuid. liquibase-update-to-latest-2.0.x.xml creates a new record in the concept_datatype table with uuid=NULL.
The CIEL sql script is executed before liquibase-update-to-latest-2.0.x.xml which results in a failing build.
This is fixed by using the 2.3.x snapshot and the 2.4.x update change logs in openmrs-standalone. This can be achieved by a change in openmrs-core 2.4.x and master.
c) Out-of-memory exceptions when loading large data sets
openmrs_concepts_1.9.9_20171118.sql and Demo-1.9.0.sql are about 110 MB each and applying these sql files via Liquibase can result in out-of-memory exceptions.
A possible fix for this is to split the sql files into multiple smaller sql files.
Next steps
- First update openmrs-core as described under (b) above
- Next update openmrs-standalone to use Liquibase 3.10.2
- Try building a release for openmrs-standalone
- If needed split the large sql files into smaller ones
I implemented the changes needed for a) and b) and still need to run a couple of tests on my side before creating the pull requests.
Please let me know which timelines you have in mind for the next release of openmrs-standalone. I am not available between Oct 12 and 16.
I could create the pull requests before Oct 12. That would give you time to test the changes between Oct 12 and 16. If further changes are needed I could look into them after Oct 16.
Please let me know what you think.
Cheers, Wolf
Am running the command on the master branch
hello @wolf am going to build the release for openmrs-standalone with the new changes made and then will share the feedback here
@wolf i have a build but its failing OpenMRS Platform - OpenMRS Platform - 2.4.x 32: Build result summary - OpenMRS Bamboo
|05-Oct-2020 09:49:50|[ERROR] Migration failed for change set /home/bamboo-agent/bamboo-agent/xml-data/build-dir/OP-OPM8-BS/standalone/target/liquibase/liquibase-ciel-data.xml::20120315-1000::standalone:|
|---|---|
|05-Oct-2020 09:49:50|[ERROR] Reason: liquibase.exception.DatabaseException: Unknown table 'concept_set_derived' [Failed SQL: (1051) DROP TABLE concept_set_derived]|
That looks like you are not using yesterday’s pull request for openmrs-standalone yet. The PR extracted the table concept_set_derived from change set 20120315-1000.
Please merge this PR first: https://github.com/openmrs/openmrs-standalone/pull/51
thanks @wolf for the feedback,
however i dont have the rights to merge PRs in core ,we can wait for @dkayiwa to do the merging and then i will buid again and provide feedback here
@wolf i have done a ci build after @dkayiwa merging the above PR and got https://ci.openmrs.org/browse/OP-OPM8-35
|05-Oct-2020 17:27:44|[INFO] Marking ChangeSet: /home/bamboo-agent/bamboo-agent/xml-data/build-dir/OP-OPM8-BS/standalone/target/liquibase/liquibase-ciel-data.xml::20201003-1000::standalone ran despite precondition failure due to onFail='MARK_RAN': |
|---|---|
|05-Oct-2020 17:27:44| /home/bamboo-agent/bamboo-agent/xml-data/build-dir/OP-OPM8-BS/standalone/target/liquibase/liquibase-ciel-data.xml : Table concept_set_derived does not exist|
This is good news, it shows that the build can handle the absence of the table concept_set_derived which was dropped by OpenMRS version 2.0.
The next step is to import the CIEL concept dictionary.
|05-Oct-2020 17:27:45|[INFO] INSERT INTO liquibasechangelog (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, `DESCRIPTION`, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('20120315-1000', 'standalone', '/home/bamboo-agent/bamboo-agent/xml-data/build-dir/OP-OPM8-BS/standalone/target/liquibase/liquibase-ciel-data.xml', NOW(), 901, '8:842f411a0f413d97a8aae085bdd90c95', 'sql; dropTable tableName=concept; dropTable tableName=concept_answer; dropTable tableName=concept_class; dropTable tableName=concept_complex; dropTable tableName=concept_datatype; dropTable tableName=concept_description; dropTable tableName=concep...', 'Preparing schema for adding ciel data', 'EXECUTED', NULL, NULL, '3.10.2', '1918842075')|
|---|---|
|05-Oct-2020 18:10:59|Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.|
The timestamps between these two log messages show that the import has been running for 43 minutes. It looks like Bamboo considered this to be a hanging build and gave up.
A couple of suggestions:
-
Could you please turn off the force stop build feature and configure Bamboo to run the maven command with the -X parameter, this will give us the full log (it is going to be seriously large).
-
I never succeeded in loading the CIEL data locally, it just took to long or Maven process was stopped by an out-of-memory exception. I tried to split the CIEL data set into a series of smaller sql files which looked promising but takes a lot of time to implement and I did not finish this.
We can decide about splitting the file after we got the more detailed log.
Before the liquibase version upgrade, it never took this long. Could this be a problem with liquibase?
Hmm, that is a good question. Let me run a couple of tests locally, I can tweak the standalone build so that the only thing it does is loading the CIEL data and use both Liquibase versions for that.
1 Like
Good thinking regarding the liquibase update, Daniel.
I ran a few local tests with Liquibase 2.0.1 and 3.10.2:
- Liquibase 2.0.1 loads the full CIEL dataset in 25 seconds
- Liquibase 3.10.2 needs more than 14 minutes, at that point I terminated the process
I could see two ways out of this situation:
- Talk to the Liquibase people if they are aware of the problem and if there is a fix
- Split the openmrs-standalone build into multiple parts that use different versions of Liquibase. This is a hack but it may be the best option we have unless Liquibase can provide a fix.
What do you think?
1 Like
@wolf thanks for the feedback and we can raise this to the liquibase team and see what they have to say and if no positive feedback from them then we can apply the hacky plan B
1 Like
I would go for this, because we eventually plan to drop the standalone and replace it with something dockerish.
Actually this turns out not to be a hack, we can use Maven modules for that and it works quite nicely. I am working on it.
1 Like
Sounds like music in my ears! Thanks @wolf 
1 Like
that is good to hear, @dkayiwa.
Most of development is done and a few more integration test runs are needed.
1 Like
thanks @wolf for the magic u are applying to resolve this
1 Like