GSOC 2023: o3-Draw-on-body-diagram app- Project Updates & Discussion

Hello folks :wave:,

I will be working on the project o3:Draw-on-body-diagram for the OpenMRS community with very high priority for orgs like @ICRC and many more.

The main objective of this project is to improve the diagramming feature in OpenMRS, a medical record system, by allowing the upload of any diagram as an image, annotating certain areas of the diagram with different shapes, saving and retrieving these diagrams, and downloading them as an image with edited annotations. The project has two parts, essential and desirable objectives(more to come in the further implementation), with the former providing basic functionalities for uploading and editing diagrams while the latter enhances the user experience by adding more features such as drawing free shapes and downloading annotated diagrams for easy sharing and printing.

This thread will be used for project updates and discussions.


Primary: @heshan :raised_hands:

Backup : @jayasanka :raised_hands:

Student: @thembo42 :innocent:


Project Details(still being worked on): Link

The ICRC war surgery manual : Link

Istanbul protocol to document torture/ill treatment: Link

All technical thoughts are welcome!

@grace @jesplana @pauladams @cduffy @mksd @ibacher @ball @johnblack @mogoodrich @burke @jayasanka @kdaud @heshan


FIRST MENTOR MEETING :partying_face: :partying_face:

When: 2023-05-12T12:00:00Z

Where: Google meet

Link: Gsoc mentor meeting Friday, May 12 Time zone: Africa/Nairobi Google Meet joining info Video call link:


  1. Pro Tips and Guidance

  2. Project Timeline(realistic break down)

  3. Technologies Required(update a few things)

  4. UI/UX Designs(moving forward, I will love to hear from the @ICRC team)

@heshan @jayasanka @kdaud @jesplana and all that are interested :partying_face:


Some random thoughts:

  • Avoid building a body-specific solution when an image notation solution that can use a body could also be used for notation of other clinical images
  • Provide methods for defining regions and then answering questions like “what notations are in this region” and “which regions include this notation”
  • Design a portable solution that can be introduced where needed in the app rather than making assumptions about where the diagram tool will be used.
  • Make it really easy to add a simple notation, since the most common use case for diagrams like this are one or two notations (e.g., show where a specific finding is located with detail provided elsewhere in a clinical note)

Thanks so much @burke , that was insightful,

Just if I got you right(you have detailed; portability, flexibility and scalability)

  1. Body-specific vs. general image notation: To avoid building a body-specific solution, the diagramming feature should be designed in a way that can be used to annotate any type of clinical image, not just body diagrams. This would require designing a flexible annotation tool that can be applied to different types of images.

Some general questions that can arise to make a more reliable solution and get a well rounded and grounded understanding of the whole project. Feel free to drop your thoughts. :hugs:

  1. How will the diagramming feature be integrated into the existing OpenMRS system? Will it be developed as a standalone micro-frontend or integrated into an existing module? How will it interact with the Java and Spring components of the core OpenMRS system?

  2. What image formats will be supported for uploading and annotating images? Will the diagramming feature be able to handle large images, such as high-resolution medical images?(addressed some how in the proposal)

  3. How will annotations be stored and retrieved from the system? Will they be stored as part of the patient record or in a separate database? How will annotations be associated with specific patients and encounters?

  4. What user roles and permissions will be needed for the diagramming feature? Will different roles have different levels of access or permissions for creating, editing, or viewing annotations?

  5. How will the diagramming feature be tested and validated(both ui/ux and the system functionality), both during development and after release? What metrics will be used to measure the success of the feature?

  6. What overall tools and libraries will be used for developing the diagramming feature? Will they be compatible with the existing OpenMRS technology stack?

  7. How will the project be scoped and prioritized, given the constraints of the GSoC timeline and the size of the project? What features or functionality should be considered essential versus desirable, and how will they be implemented and tested?

Some insights were shared in the proposal but now we can scope it down to the needs of openmrs and other orgs. I suppose answering these will get me up to speed for the next phase.

@heshan @jayasanka @ibacher @burke @grace @jesplana

Previously I had come up with an epic that mainly was tailored to refactoring the legacy drawing module, I know refactoring is quite expensive.

I suppose i need guidance whether to continue with refactoring or get a fresh module(I know @heshan & @jayasanka will guide me here)

This partly answers question one;

Portable solution: To ensure the diagramming feature can be introduced where needed in the app, it should be designed as a portable component that can be integrated into different parts of the OpenMRS system as needed. This might involve using standard APIs or designing a modular architecture that can be easily customized.


On May 12, 2023 3:00 PM i had my first mentor meeting and it was insightful with the agenda covered and mainly focusing on leveraging the workflow of uploading an annotated diagram with the best data structure and api implementation. With curiosity i first looked at the GitHub - openmrs/openmrs-module-attachments: UI components and backend web services to upload, view and manage attachments within OpenMRS. “Attachments” module that provides functionality for uploading and managing file attachments, including images.

Testing it out, it lacked some APIs(though it has most functionalty, will need guidance on how to still use it to avoid re-inventing the wheel @heshan @ibacher @dkayiwa @jayasanka ). I know I can Integrate the necessary APIs from the Attachments module into this project to enable image upload functionality but i think we can look at the new implementations then advise accordingly


                     **BACK-END FOCUS**

I wanted to provide us with an update on the progress I’ve made so far on the scoped-down solution we discussed with @heshan .This week, the focus has been/is on allowing users to upload annotated images, storing them in the database, and implementing the necessary APIs for storage, retrieval, and display on the frontend.

Here are the details of the implementation:

  1. Database Implementation:
  • Created the required tables in the database to store image files and annotations.
  • The Images table includes columns for filename, filesize, filetype, metadata, and data (LONGBLOB type).

Screenshot from 2023-05-16 18-01-46

  • The DiagramAnnotations table includes columns for diagram ID, x and y coordinates, and description.

Screenshot from 2023-05-16 18-03-04

  1. API Implementations:
  • Implemented the necessary API endpoints for handling image upload, retrieval, storage, and download.
  • The ImageController class includes methods for handling HTTP requests related to image operations.
  • The endpoints accept the necessary parameters, such as file uploads, and interact with the corresponding service methods.
  1. Service and Repository Implementations:
  • Implemented the ImageService class responsible for handling the business logic of image operations.
  • Implemented the ImageRepository class to interact with the database and perform CRUD operations on images.

Preparing a demo for this

Please let me know if there are any specific aspects you would like me to focus on or if there are any additional requirements or suggestions you have for the project. I’m looking forward to your feedback.

Thank you

Hi @burke and @thembo42 , please see my comments below:

  • Avoid building a body-specific solution when an image notation solution that can use a body could also be used for notation of other clinical images
    Could we select from a template? For example, select the diagram name from a drop down, then the diagram loads. Once loaded, then we can start drawing or add notation. Additionally, the list from the drop down can also be based on the gender and other variables.

  • Provide methods for defining regions and then answering questions like “what notations are in this region” and “which regions include this notation”
    → Could we then have the region question in the notation pop-up window? Also, could we pre-filter the diagram templates based on the region? For example, the user selects “Lower Limb (LL) - Right” in the region, then the diagram is filtered to show only diagrams? But i’m not if this can be saved in the table (i.e. obs_group to store region info, value_text to store the notation, etc.?)

1 Like

sure this makes perfect sense. Some thing similar you shared some time back @jesplana

@burke does this make it possible to have varried clinical images and i suppose the data structure will handle the different diagrams . Did i get this right?

i suppose its possible one can display a prompt or a form field asking them to define the region. This way, when the notation is saved, it will be associated with the corresponding region information.

i suppose this is too possible, just to refer to the table i defined above DiagramAnnotations table, I would suggest to add columns region_id or region_name to store the region information associated with each annotation.

Needs quite a bit of modification though. Am not sure if i got that well!

Quite confusing to get the right data structure sufficient here. the scope seems tricky :face_with_open_eyes_and_hand_over_mouth: @ibacher @heshan @jayasanka can we scope out this am in a valley of whether to setup the project with the Attachment module apis and make modifications(am skeptical of the cost) or i set up fresh structure and apis(afraid of re inventing the wheel)


When: 2023-05-19T12:00:00Z

Where: Google meet

Link: 03-Daw-On-body meetings Friday, May 19 · 3:00 – 4:00pm Time zone: Africa/Nairobi Google Meet joining info Video call link:


  1. updates on weeks tasks
  2. Tips and guidance moving foward
  3. AOB

@heshan @jayasanka @kdaud @jesplana and all that are interested :partying_face:



Quite abit of tasks and learning areas have gone through the last week. The main focus has been;

  1. Organizing the software engineering thought process around building this o3 app. my mentor @heshan took me through a scoped down solution focusing on dividing the system into manageable pieces.


Looking at the data structure of my 03 application especially the diagramming and annotating structure, I was challenged to visualize what users will expect to see on the front end and then structure needed to incarnet this i.e User Centric Design. So there was need to update the previuosly designed data base structure to include more design anthetetics;

Diagram table;

DiagramAnnotations table;


  • Looking at the entire UI/UX and scoping down to ;
  1. Visualizing the list of diagrams created

  2. Create diagram with annotation visual workflow

  3. Upload diagram visual workflow

  • APIs for;
  1. Pick diagrams from db to the front end
  2. Array of diagrams
  3. Create diagram
  4. Delete diagram
  5. Replication of the diagrams

Further Testing the APIs of the GitHub - openmrs/openmrs-module-attachments: UI components and backend web services to upload, view and manage attachments within OpenMRS.

@jayasanka you could guide me on leveraging this REST API

Getting a referesher in spring

Thanks @heshan and @jayasanka

cc/ @grace @jesplana @burke @ibacher @reagan @mozzy

1 Like


A few questions I used to gather requirements that i hope would address this week’s tasks(Annotated Diagram upload);

  • in what situation do they need to draw something on the body in an EMR? During the admission or assessment process, doctors will indicate which part of the body was wounded or need a specific operation.

    Screenshot from 2023-05-26 10-19-19

  • what number one thing do you want to do on the screen especially editing/anoatating screen

  1. First, the user needs to be able to fill in a form related to the admission or the assessment being conducted,

    Screenshot from 2023-05-26 10-26-10

  2. then when necessary, choose a template to document the injury or body parts involved for the intervention.

    Screenshot from 2023-05-26 10-27-45

  3. draw on the image

    Screenshot from 2023-05-26 10-28-33

  4. image is saved as part of the form

  5. Be able to view all of the images/attachments for the patient Screenshot from 2023-05-26 10-29-51

    Screenshot from 2023-05-26 10-32-17

  6. Do a side-by-side comparison (but this can be done by viewing in a new window)

Screenshot from 2023-05-26 10-34-25

Screenshot from 2023-05-26 10-38-30

Request: GET /diagrams


    "id": 1,
    "filename": "diagram1.png",
    "filesize": 1024,
    "filetype": "image/png",
    "description": "Pain measurement",
    "created_at": "2023-05-17T10:30:00Z",
    "updated_at": "2023-05-17T10:30:00Z",
    "annotations": [
        "id": 1,
        "diagram_id": 1,
        "x_coordinate": 100,
        "y_coordinate": 200,
        "description": "saures",
        "created_at": "2023-05-26:45:00Z",
        "updated_at": "2023-05-26:45:00Z"
        "id": 2,
        "diagram_id": 1,
        "x_coordinate": 300,
        "y_coordinate": 400,
        "description": "stars",
        "created_at": "2023-05-26:00:00Z",
        "updated_at": "2023-05-26:00:00Z"
    "id": 2,
    "filename": "diagram2.png",
    "filesize": 2048,
    "filetype": "image/png",
    "description": "RTI scan",
    "created_at": "2023-05-26:15:00Z",
    "updated_at": "2023-05-26:15:00Z",
    "annotations": [
        "id": 3,
        "diagram_id": 2,
        "x_coordinate": 150,
        "y_coordinate": 250,
        "description": "circles",
        "created_at": "2023-05-17T11:30:00Z",
        "updated_at": "2023-05-17T11:30:00Z"

More of the API implementation to be seen in week one that is next week. I want to welcome suggestions and adjustments to the designs as we prepare for week one coding period.

Thank you

@heshan @jayasanka @cduffy @pauladams @jesplana