Generating Coverage Reports using gcov

This tutorial will walk through generating coverage reports for your projects. This will help you ensure that your test suite tests just about everything in your program. Both clang and gcc have the machinery to generate coverage reports; we will use gcc in this tutorial.

Prerequisites

This tutorial is based off of the Makefiles and test directory structure used in the starter code. In particular, we will make changes to p*_files/Makefile and p*_files/p*_tests/Makefile, and we will put our test cases in p*_files/p*_tests/.

1. Creating a test suite

We first need to add files to our test suite in the p*_tests directory. The Makefile has a rule to automaticlaly run every test in the directory. Look at the test cases included in the starter code for each project to see which files the Makefile is expecting for each case. For project 4, each case needs the following files:

2. Modifying make test

p*_files/p*tests/Makefile will exit as soon as a test case fails. We do not necessarily want this behavior for coverage reporting. To ensure that make test will continue to run tests even after a failed case, we prepend '-' to the line which runs dmc on your test case.

Example for Project 4: p4_tests/Makefile:

TESTFILES := $(wildcard *.dm)
TESTS := $(TESTFILES:.dm=.test)

.PHONY: all

all: $(TESTS)

%.test:
	@rm -f $*.unparse $*.err
	@touch $*.unparse
	@echo "TEST $*"
	# The '-' tells make to carry on even if this command exits with an error
	-@../dmc $*.dm -u $*.unparse 2> $*.err ;\
	PROG_EXIT_CODE=$$?;\
	if [ $$PROG_EXIT_CODE != 0 ]; then \
		echo "dmc error:"; \
		cat $*.err; \
		exit 1; \
	fi; \
	diff -B --ignore-all-space $*.unparse $*.unparse.expected; \
	STDOUT_DIFF_EXIT=$$?;\
	exit $$STDOUT_DIFF_EXIT || echo "Tests passed"

clean:
	rm -f *.unparse *.err

3. Invoking g++ with --coverage

The --coverage option tells g++ to instrument the resulting binary for coverage analysis. We can add this option to the FLAGS variable in the top-level Makefile.

Example for Project 4: p4_files/Makefile:

FLAGS=--coverage -pedantic -Wall -Wextra -Wcast-align -Wcast-qual -Wctor-dtor-privacy -Wdisabled-optimization -Wformat=2 -Wuninitialized -Winit-self -Wmissing-declarations -Wmissing-include-dirs -Wold-style-cast -Woverloaded-virtual -Wredundant-decls -Wsign-conversion -Wsign-promo -Wstrict-overflow=5 -Wundef -Werror -Wno-unused -Wno-unused-parameter

4. Running tests

We are now ready to build the instrumented binary and run our test suite. From the project directory, run:

make clean
make test

Running the tests will cause the binary to generate some data in *.gcda files. This is where the coverage information comes from.

5. Generating and viewing the coverage report

Now we can generate and view the coverage reports for source files we're interested in using gcov. For example, if we are interested in unparse.cpp, run the following command in the project directory:

gcov unparse

This command prints the percent coverage in unparse.cpp and included files to the standard output. It also creates the file unparse.cpp.gcov, which has the counts of how many times each source line is executed throughout the test suite.

6. Updating `make clean`

We're done instrumenting our binary, now let's clean up. Add *.gcno *.gcda *.gcov *.d to the clean rule.

Example for Project 4: p4_files/Makefile:

clean:
	rm -rf *.output *.o *.cc *.hh *.gcno *.gcda *.gcov $(DEPS) dmc parser.dot parser.png