How to enable code coverage on the umtsmon module tests

Introduction

The umtsmon project contains several module tests in the umtsmon/test directory. Of course, it would be interesting to see how much of the code actually gets covered by these module tests. This page explains how to use gcov (see its man page) to do code coverage analysis.

Preparations

In order to enable code coverage logging, you need to compile all code again with specific extra compiler flags. These are already mentioned in test/GenericTestPro.inc but are outcommented in CVS:

### uncomment these two lines for compiling with code coverage instrumentation
#QMAKE_CXXFLAGS_DEBUG += -fprofile-arcs -ftest-coverage
#QMAKE_LFLAGS_DEBUG += -fprofile-arcs -ftest-coverage

Compilation

Now we need to run QT3's qmake and make to actually recompile all source code. Note that all required source files will be recompiled: the location changed, remember?
Anyway, we now have the final executable again, but instrumented. And the whole directory is filled with object files...

Running

This is the easy part: just run the application as usual. It will create *.gcda and *.gcno files for each source file or include file. Of course, the execution speed drops significantly due to all the logging it needs to do.
Then we run gcov on the main source file to generate the reports. The following are some examples for files in different source directories:

   gcov NetworkChangerTest_main.cpp
   gcov ../../src/controller/NetworkChanger.cpp
   gcov ../../src/model/Query.cpp
   gcov ../../.ui/networkoperatorwizard.cpp

Analysis

Let's look at a fragment of such file:

        8:  108:Query::ReturnCode Query::run()
        -:  109:{
        -:  110:        // safety net
        8:  111:        assert(theSerialPortPtr != NULL);
        8:  112:        if (theSerialPortPtr == NULL)
        -:  113:        {
    #####:  114:                return NO_DEVICE;
        -:  115:        }
        -:  116:
        -:  117:        // more safety net: lock the mutex
        8:  118:        assert(hasLock == false);
        8:  119:        if (theQueryMutex.tryLock()==FALSE)
        -:  120:        {
    #####:  121:                hasLock = false;
    #####:  122:                DEBUG5("Query instance %p trylock FAILED\n", this);
    #####:  123:                return NO_LOCK;
        -:  124:        }

Viewing the above output, it shows that all special situations, like error conditions, have not been executed. That is usually unwanted behaviour - a good test case excerts both good and bad behavior. Browsing through results, you'll notice that not all lines have been executed. gcov actually tells how much lines were touched during the run:

File '../../src/base/SerialPort.cpp'
Lines executed:25.29% of 174        
../../src/base/SerialPort.cpp:creating 'SerialPort.cpp.gcov'

Advanced Topics

Of course, some files are exerted by several tests - a good example is the Query class, which is used by the majority of the test cases, and each test case touches diffent parts of the code. gcov by itself is capable to add data of multiple runs into the same graph files - allowing to run all tests and only then generate reports.
The easiest way to do this, is by copying all of the sources into a single new directory in the test directory and then qmake,make,run the first test, then qmake,make,run the second test, qmake,make,run the third test, etc. Once all runs are finished, you can then run gcov on the source file.

This way, running both NetworkOperatorWizardTester and NetworkChangerTester, you get a code coverage of 81.7% on NetworkChanger.cpp, 70.8% on networkoperatorwizard.ui.h and 97.3% of OperatorList.cpp. That's good.
Another interesting example is Query.cpp. After running the above two tests, 77.4% of the code of Query is touched. Adding the QueryTester then improves the coverage of Query to 78.3%.