Will try it out, for performance comparison I think the Bahmni CSV import can be used with multiple threads, to see how it perform.s
Thank you so much @mihirkh!
If you can also set up some sort of OpenMRS instance demonstrating the differences between the two, that will greatly help in evaluating to make an informed decision on the way forward regarding this.
Can you please share the code of the Bahmni CSV import? Using multiple threads to do it is non-trivial. It may even be the problem in fact …
This is the library called by the bahmni-core omod
This line in MultiStageMigrator.java is suspicious:
List<T> csvRowsForNextStage = readCsvFileToMemory(csvFile, entityClass);
If the CSV is huge, unless you have lots of free memory the heap will suffer (and performance too). I would refactor the process reading lines one by one. But of course this would be harder to implement (1 thread reading and a pool of threads doing the work). You can try also with a single thread.
Also the runInParallel method creates exactly 5 threads that will take 5 connections from the pool until they complete => not much work for the connection pooler. If the machine has less than 5 cores they will compete with each other to get a slice of CPU.
A good discussion here:
I’ve been looking into doing some OpenMRS performance tuning and was wondering, does anyone know if there is reason we set the idle time of our connections so low? I’m seeing this configuration:
hibernate.c3p0.timeout=100
So we are timing out idle connections after 100 seconds (I confirmed in one of our OpenMRS instance that the underlying c3p0 property maxIdleTime is 100).
From the c3p0 docs:
“Some general advice about all of these timeout parameters: Slow down! The point of Connection pooling is to bear the cost of acquiring a Connection only once, and then to reuse the Connection many, many times. Most databases support Connections that remain open for hours at a time. There’s no need to churn through all your Connections every few seconds or minutes. Setting maxConnectionAge or maxIdleTime to 1800 (30 minutes) is quite aggressive. For most databases, several hours may be more appropriate. You can ensure the reliability of your Connections by testing them, rather than by tossing them. (see Configuring Connection Testing.) The only one of these parameters that should generally be set to a few minutes or less is maxIdleTimeExcessConnections.”
Still trying to understand the details of connection pooling myself. I recently read this a couple places, but am not 100% sure I understand the reasoning behind it:
hibernate.c3p0.idle_test_periods must not be higher than hibernate.c3p0.timeout or connections closed by the database will not be properly detected.
In the OpenMRS standard config, idle_test_period=3000 and timeout=100.