openmrs-core test coverage dropped after moving to github action

teleivo · July 29, 2021, 12:57pm

Hi,

I noticed that the test coverage of openmrs-core is reported to be around 10% by coveralls after being at around 65%. You can see a nice history of the test coverage at openmrs/openmrs-core | Coveralls - Test Coverage History & Statistics

I am pretty sure it’s because we run only the integration tests as you can see here

github.com

openmrs/openmrs-core/blob/master/.github/workflows/build.yaml#L45

    
      
              java-version: ${{ matrix.java-version }}
          - name: Cache Maven packages
            uses: actions/cache@v2.1.6
            with:
              path: ~/.m2
              key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
              restore-keys: ${{ runner.os }}-m2
          - name: Install dependencies
            run: mvn clean install -DskipTests=true -Dmaven.javadoc.skip=true --batch-mode --show-version --file pom.xml
          - name: Build with Maven
            run: mvn test -Pskip-default-test -Pintegration-test --batch-mode --file pom.xml
          # this is necessary to populate the environment variables for Coveralls properly
          - name: Set branch name and PR number
            id: refs
            if: ${{ matrix.java-version == '8' }}
            continue-on-error: true
            env:
              BRANCH_NAME_OR_REF: ${{ github.head_ref || github.ref }}
            run: |
              echo "::set-output name=branch_name::${BRANCH_NAME_OR_REF#refs/heads/}"
              echo "::set-output name=pr_number::$(jq --raw-output .pull_request.number "$GITHUB_EVENT_PATH")"

I remember we had this issue when running/reporting the test coverage from Travis CI. See issue/fix from then Keep test results for reporting coverage · openmrs/openmrs-core@02eab6a · GitHub

@ibacher do you know why we only run the integration tests in the build step?

dkayiwa · July 29, 2021, 8:07pm

I have just added back the running of the other tests. That was a very good catch @teleivo!

ibacher · August 2, 2021, 2:32pm

Thanks for the catch. That was an error on my part. The pattern I was following was to install dependencies and then build the project. I really should’ve had three steps there:

Install dependencies
Build and test the project
Run the integration tests

But there’s no reason 2 & 3 can’t be done at the same time.

teleivo · September 29, 2021, 4:25am

I think there are

If we run them separately instead of sequentially

we can run them in parallel, making test runs faster.
if some unit tests fail, we would also see if an integration test fails. This helps a developer fix all tests before pushing again.

Collecting the aggregate code coverage might be more involved than before though.