Great conversation! Thanks for this post, @aojwang.
I favor method 3 (hybrid approach). More specifically, I wouldn’t try to design an ETL solution that meets every need out of the box; rather, I’d suggest something like:
- Create a docker-based solution
- Create a process for defining a transform that the solution will generate & update. Preferably, configuring existing Apache tools.
- Use that process to define some useful transforms (like flattened encounters).
- Add an engine to automatically use this process to auto-flatten forms.
- As a proof of concept, use this approach to generate the transforms needed for PEPFAR report(s) that everyone could use.
Then, implementations could adapt the solution to meet their needs by adding transforms the same way we defined the transforms that are included out of the box and we are relieved of the burden of needing to define every transform needed in the community.