This is great @dkayiwa - very excited about this initiative.
I think you are aware of these issues, but a few initial thoughts:
- Handling metadata changes. Let’s say after the system has been running for some time, the implementation team decides to change the name of one of their concepts or locations or encounter types. What happens to the denormalized data at that point? Is this something the design would be able to handle?
- Similar to #1, let’s say a bug has been found in a transformation. Or a new column has been added to a table in the core data model. Or some other changes are decided upon after the representations have been built? What would be the process for deciding what needs to be rebuilt and what are the implications if rebuilding this takes a long time and production level decision support rules and such are relying upon it?
- Handling direct database changes. The event module is getting better, and the work that @raff is involved with and which is getting discussed in this thread and this thread seem like they will continue to make events more reliable. But these are still only as comprehensive as the mechanisms that detect changes and fire events - generally via Hibernate interceptors. There are still plenty of situations where the database is updated directly, most notably when liquibase is used to perform database migrations or setup initial metadata. How will these situations be handled such that the query store maintains consistency?
- Would you expect modules that do analytics ETL like mamba to migrate to using this mechanism?
Thanks! Mike