The process of manually adding concepts to concept set is straight forward. But I have 1000s of concepts that comes under a concept class and I need to get them organised into concept sets. For example, 1000s of concepts comes under diagnosis concept class. From these, I need to move respiratory diagnosis to a concept set named respiratory diagnosis.
Is there any way I can do that other than using manual procedure.
Thanks very much
You can write a script to do this, either using the OpenMRS REST API, using the Groovy module (and behind it, the Java API), or you could do direct SQL queries against the database.
You will have to actually write the script though.
I would suggest that you write a sql script to do this easily, but it will require that you know or can get the concept ids. You can write a select statement to add all the concepts with a class = diagnoses, and then remove all the non-respiratory diagnoses, but I am guessing you have many diagnoses.
First create the concept set (use the ui since this is easy). Class is ConvSet; Datatype is N/A, Checkbox selected for set. For this example, the concept_id for this concept (1234) (concept_set), the creator of ‘1’ is the admin user, and the respiratory diagnoses are concept_id 101, 102, etc :
insert into concept_set
(concept_id,concept_set,creator,uuid)
values
(101,1234,1,uuid()),
(102, 1234, 1, uuid()),
…
;
First, why are you doing this? Are you creating a maintenance nightmare for yourself? Should you be leveraging SNOMED mapping for example, or one of the existing CIEL concept sets for common ICPC diagnoses? We can help, but would be better to understand what you are actually trying to accomplish…
Thanks Ball … you are amazing. I will review the process and let you know.
OK @akanter … The main reason is to allow providers to add coded diagnosis through using the diagnosis widget in Visit Note form. We have the required concepts but that are not included under the Concept set: ICPC diagnosis categories. Therefore, providers cannot search for them. I can add them manually, but as you described it, it is a nightmare. There are 10s of thousands of diagnosis concepts. I can create a concept and autocomplete function to search for concepts in a given class. But the problem is that it will be part of a form and it would not show as nice on dashboard and visit page as the current diagnosis widget. I hope this makes sense.
Any suggestion is highly appreciated 
There are different ways to address this issue. Are your providers transcribing from printed forms or are they entering data directly into OpenMRS at point of care? If the latter, then you really need to allow for full diagnosis look up from the database. One way we handled it in MVP was to have a set of frequently used diagnoses as check boxes and these were added to visit diagnosis or problem added array. Then there was a search which allowed for the database search for any other unlisted diagnosis. @darius do you know what Yani is trying to accomplish?
Thanks @akanter and @darius Yes the data is collected at point of care and entered directly on OpenMRS. That’s why I am searching for a method to add all diagnosis to ICPC diagnosis categories. What we did is adding diagnosis as we go. However, it is time consuming.
Hello @ball I will probably try your approach. but I cannot collect all concept ids. So I will create a new set called general diagnosis. and add all concepts with class=diagnosis under it. would that work you think. how that will look on the scrip you suggested above .
Thanks very much Ball
Thanks for all your help on this @darius @dkayiwa @akanter @ball provided great guidance and I moved the concepts (37,000 of them) under once concept set using MySQL script. However, when I login to concept dictionary from the UI, and try to view my concept set (I named it general diagnosis), I get the following error
An error has occurred! ** The following error happened somewhere on this page:** ** java.lang.OutOfMemoryError: Java heap space ** ** (The full error stack trace output is in the source of this page.)**
In the logs I found this
09-Oct-2017 16:30:41.242 SEVERE [http-nio-80-ClientPoller-0] org.apache.tomcat.util.net.NioEndpoint$Poller.run java.lang.OutOfMemoryError: Java heap space
vemq.broker.region.RegionBroker$1.run(RegionBroker.java:110) at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) at java.util.TimerThread.mainLoop(Unknown Source) at java.util.TimerThread.run(Unknown Source) Exception in thread “C3P0PooledConnectionPoolManager[identityToken->1bquygx9ry7bku18c7orj|49ce6d53]-AdminTaskTimer” java.lang.OutOfMemoryError: Java heap space SEVERE:Memory usage is low, parachute is non existent, your system may start failing. java.lang.OutOfMemoryError: Java heap space SEVERE:Memory usage is low, parachute is non existent, your system may start failing. Exception in thread “ActiveMQ Journal Checkpoint Worker” Exception in thread “C3P0PooledConnectionPoolManager[identityToken->1bquygx9ry7bku18c7orj|49ce6d53]-HelperThread-#1” Exception in thread “AsyncFileHandlerWriter-1304836502” java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space
Does this means that I cannot add 37000 concepts under one concept set. Is there still a hope to make this work.
Always grateful for your help
Hi @yadamz, sorry for not really engaging with this thread earlier.
The current OpenMRS code is not written to handle tens of thousands of concepts in a diagnosis set-of-sets. In the big picture the idea of using these ICPC sets was supposed to be a temporary placeholder, and we were supposed to explore using the full CIEL dictionary’s diagnoses, but never got to this. So, to make this work, you’re going to have to write some Java code, but this is actually a desired option/feature to add to the OpenMRS Reference Application to allow the diagnosis list to be controlled either by concept sets, or by a concept class. So, I think people can help guide you in this.
Offhand I think you’d need to do the following:
I expect a bit more cleanup might have to happen after this, but would get you started…
Thanks @darius Do you know how many concepts a concept set can handle with the current code? I am thinking of creating multiple sets and divide the 37000 concept among them as a temporary solution.
Thanks again
That won’t work, because the ultimate set of sets is still 37k.
I don’t know what the limit is, as I don’t think anyone has stress tested the current code. As a wild guess maybe 5k?
Maybe @ball can say how many diagnoses PIH have in their largest list.
-Darius (by phone)
The largest set (Surgery diagnoses) has 393 members. We have sets for sets but with a total of 1,778 diagnosis concepts. This is an example of our set of sets:
You might not hit this, but just in case. We have also hit a bug with the limit of concepts in the htmlform entry module but hope it can be fixed. We have a temporary workaround. [HTML-658] Exceeding limit on URL with too many answers within answerConceptSetIds - OpenMRS Issues
Ellen Ball | Partners In Health
Thanks @darius and @ball As I mentioned before, I added the 37000 diagnoses under one set (I called it General Diagnosis set) and I added this set to ICPC diagnosis. Now, when I try to add a diagnosis using visit note form, I can search through all 37000 with no issue. I can select and add any diagnosis. The issue I am stuck with is when I try to view/edit the General Diagnosis set from the UI, I get out of memory error. Please note that when I check the ICPC set of sets which include the General Diagnosis set with 37000 concepts, I have no problem viewing/editing it. Then, my assumption is that the problem can be avoided if I create let’s say 10 concepts sets with 3700 concepts in each. I will test this approach and report back.
@ball I might need your help again to adjust the script to randomly add the 37000 concepts into 10 concepts sets (3700 in each concept set) do you think this is doable using MySQL script?
