MambaETL v1.0.0 Released! 🎉

Software O3
,
1

Hello OpenMRS Community,

Exciting news! The MambaETL core Release v1.0.0 is now out :rocket:. A big thanks to all the contributors: Derrick Baluku (UgandaEMR) and the UCSF team (Laureen Omare, Amoso Laboso, Arthur D. Mugume, Eudson Bambo) for their invaluable input.

What is MambaETL? MambaETL (or Mamba) is designed as an OpenMRS implementation to streamline data Extraction, Loading, and Transforming (ETL) into a denormalized format, addressing the challenge of OpenMRS’s long-format storage that results in the Obs table rapidly amassing millions of records per encounter, thereby impeding efficient data retrieval and analysis in medium-sized facilities.

:link: Project Link: GitHub

Highlights of Release v1.0.0

  1. Automated Flattening of obs/encounter data: Flatten obs/encounter data for a more accessible wide data format, simplifying reporting tasks.
  2. Automated Flattening of Obs Groups: Extends automated flattening to obs groups for improved reporting.
  3. Separate Reporting Data: Configure whether reporting data is stored in the same schema as OpenMRS transactional data or a separate schema.
  4. MambaETL Reporting API: Introducing an extra channel to fetch MambaETL reports from the reporting schema via a configurable endpoint.
  5. Database Compatibility: Extensively tested on MySQL v5.7 and above, MariaDB, supporting only SQL-compliant reporting queries.
  6. Fast Build with Template Module: A provided template/reference module allows you to set up your reporting module or add MambaETL support to your existing infrastructure, in under 15 minutes.
  7. Deployment Through Liquibase: Run a single Maven command (mvn clean install), and Mamba prepares an SQL-compliant build script containing all reporting data ready for deployment on your target reporting schema.
  8. Improved Performance: Significant focus on performance improvement, showcasing faster ETL times on large datasets (up to 11 million obs).

How Can You Contribute?

As we finalize steps to transition this work to OpenMRS community repositories, we invite you, especially those passionate about reporting and ETL, to participate in testing.

Check out the quick start reference module on GitHub

Explore detailed instructions for implementers in the readme file and dive into the comprehensive MambaETL Documentation.

Let’s discuss and revolutionize OpenMRS reporting together! :speech_balloon: #MambaETL openmrs #DataOptimization

cc: @eudson @gomare @dbaluku @amugume @grace @mwaririm @wamz @OHRI @METS @UCSF

7 Likes
2

Thanks to all involved in the development of this and sharing it with the community! When do you plan to publish it on? addons.openmrs.org

3

We would be more than happy to do so, is there a guideline to do this or we just publish what we have as is? Also @grace maybe this would be a great opportunity for a review and move the repo into OMRS GitHub organisation.

3 Likes
4

The instructions can be found here: Module Release - Documentation - OpenMRS Wiki

1 Like
5

Hello @dkayiwa so about publishing to the add-ons. We have already published the MambaETL library to the public OpenMRS repositories. This one is a library meant to be depended on in the implementer module. Now, for the MamabaETL reference module, there is no need to publish it to the add-ons since it is only a starter module and every implementer needs to customise it to their needs before deploying.

6

So does this mean that to use MambaETL, i have to create my own new module, which will include the MambaETL dependency?

7

@dkayiwa sorry for the late response (missed this). Yes, you can create your own new module or leverage the starter module and modify that one to your specific needs (it already includes the necessary mamba dependencies).

8

For a non technical implementation team which just wants to download an omod and use, what would they have to do to get this omod?

9

@dkayiwa for such folk they can use the starter/vanilla module I shared above without any modifications (as is). So, maybe we can deploy its omod to the OpenMRS addons. We plan for a new release next week, we can plan to add it to the OpenMRS addons as well.

10

That is exactly what i was driving you towards! :slight_smile:

1 Like
11

Yes @dkayiwa we are now on the same page. Thanks