Since these are JUnit base test classes we could in JUnit 4 use a @Rule https://junit.org/junit4/javadoc/4.12/org/junit/rules/TemporaryFolder.html
JUnit 5 has no Rules anymore and solves this using https://junit.org/junit5/docs/5.4.0/api/org/junit/jupiter/api/io/TempDir.html
We currently have 2 base classes one for JUnit 4 one for JUnit 5. That is necessary so that module developers can gradually migrate. I assume the issue exists for both base classes.
We might run into the issue discussed here tests failing in core on Windows that tests start failing because JUnit is not able to delete the directory. This happens when our code or tests do not release their connection to files in the temp dir. But let’s see 