dcsimg

Analyzing Code Coverage with gcov

Before releasing any amount of code, developers usually test their work to tune performance and prove that the software works as intended. But often, validation is quite difficult, even if the application is simple.

Before releasing any amount of code, developers usually test their work to tune performance and prove that the software works as intended. But often, validation is quite difficult, even if the application is simple.

For example, the venerable Unix/Linux ls utility is conceptually quite simple, yet its many options and the myriad vagaries of the underlying file system make validating ls quite a challenge.

To help validate the operation of their code, developers often rely on test suites to either simulate or recreate operational scenarios. If the test suite is thorough, all of the features of the code can be exercised and be shown to work.

But how thorough is thorough? In theory, a completely thorough test suite would test all circumstances, validate all of the results, and exercise every single line of code, demonstrating that no code is “dead.” (As Stephen Friedl pointed out in last month’s column, dead code is a favorite hiding place for pesky bugs.) Validating results can be done in any number of ways since output is typically tangible in one form or another, but how do you make sure that all of your code was executed? Use GNU’s gcov.

Like an X-ray machine, gcov peers into your code and reports on its inner workings. And gcov is easy to use: simply compile your code with gcc and two extra options, and your code will automatically generate data that highlights statement-by-statement, run-time coverage. Best of all, gcov is readily available: if you have gcc installed, you also have gcovgcov is a standard part of the GNU development tools.

This month, let’s look at code coverage analysis and how to use gcov to help improve the quality of your code and the quality and thoroughness of your test suites.

What is Code Coverage Analysis?

As mentioned above, it’s ideal to find dead code and get rid of it. In some cases, it may be appropriate to remove dead code because it’s unneeded or obsolete. In other cases, the test suite itself may have to be expanded to be more thorough. Code coverage analysis is the (often iterative) process of finding and targeting “dead” or unexercised code, and is characterized by the following steps:

1. Find the areas of a program not exercised by the test suite.

2. Create additional test cases to exercise the dead code, thereby increasing code coverage.

3. Determine a quantitative measure of code coverage, which is an indirect measure of quality.

Code coverage analysis is also useful to identify which test cases are appropriate to run when changes are made to a program and to identify which test cases do not increase coverage.




Pity, gcov Can’t Find Logic Errors


Unfortunately, code coverage analysis doesn’t find logic errors. Consider the following code:



10: rc = call_to_xx
11: if (rc == ERROR_FATAL)
12: exit(2); /* exit with error code 2 */
13: else
14: /* continue on */

When the code coverage was checked for this snippet of code above, the output from the test coverage tool stated that line 11 was never true, so the exit with error code 2 wasn’t tested. The apparently obvious test case to write is that scenario where the operation fails with ERROR_FATAL. That seems to be sufficient — unless the call to call_to_xx routine returns other error conditions. For example, if call_to_xx returns ERROR_HANDLE, the new test would not cover the code completely.

Instead, the code should been written to handle both error conditions, ERROR_FATAL and ERROR_HANDLE.


10: rc = call_to_xx
11: if (rc == ERROR_FATAL)
12: exit(2); /* exit with error code 2 */
13: else if (rc == ERROR_HANDLE)
14: /* handle this error condition */
15: else
16: /* continued on */

The test suite should check that the code handles ERROR_HANDLE correctly. But no test does that for this error condition.

No test coverage tool will tell you that this is needed. It can’t. The test coverage tool can only identify the coverage on the code that exists.


Types of Code Coverage

There are many different types of code coverage that can be measured by gcov. To be brief, let’s discuss just two of them: branch coverage and loop coverage.

Branch coverage verifies that every branch has been taken in all directions. Similarly, loop coverage tries to verify that all paths through a loop have been tried. Loop coverage sounds complex, but actually can be verified by satisfying just three conditions:

1. The loop condition yields false so the body is not executed.

2. The loop condition is true the first time, then false, so execution of the body happens only once.

3. The loop condition is true at least two times, causing the loop to execute twice.

For example, in the following code snippet…


routine Example_function(number)
{
if (number % 2) == 0)
printf(“even \n”);
for (;number < 9; number++){
printf(“number is %d\n”, number);
}
}

…the if statement must be tested with an odd and even number. The for statement must be tested with two numbers, such that the condition (number < 9) is true and false, respectively. Therefore, the following three tests would achieve complete test coverage for the routine listed above:


Example_function(11);
Example_function(6);
Example_function(8);








compile_01
Figure One: The steps to create gcov output

gcc Options Needed for gcov

Before programs can use gcov, they must first be compiled with two gcc options: -fprofile-arcs and -ftestcoverage. These options cause the compiler to insert additional code into the object files. Then, when the code runs, it generates two files, sourcename.bb and sourcename.bbg, where sourcename is the name of your source code file.

The .bb file has a list of source files, functions within the the file, and line numbers corresponding to each block in the source file. The *.bbg file contains a list of the program flow arcs for all of the functions. Executing a gcov-enabled program also causes the dumping of counter information into a sourcename.da file when the program exits.

gcov uses the *.bbg, *.bb, and *.da files to reconstruct program flow and create a listing of the code that highlights the number of times each line was executed. Let’s try using gcov.

Compile the file sample.c shown in Listing One with the options -fprofile-arcs, -ftest-coverage, and -g.


% gcc -fprofile-arcs \
-ftest-coverage \
-g sample.c \
-o sample


Now we’re ready to see how much coverage each test case provides. Run the sample application with input of 1000.


% ./sample 1000

The application displays “Creating an 1000 by 1000 array,” and creates a new file called sample.da. Next, run gcov on the source code (if your application has more than one source file, run gcov on all of the source files)…


% gcov sample.c

gcov emits “69.23 % of 26 source lines executed in file sample.c.” This gcov command also creates the file sample.c.gcov, shown in Listing Two. In the listing, a ###### marker indicates that the associated line of source code hasn’t been executed.

Next, run the sample program with no input.


%./sample

The application displays “Usage: ./sample Enter arraysize value.” Next, run gcov again.


% gcov sample.c

76.92 % of 26 source lines executed
in sample.c


Now run the sample program with the parameter 0.


%./sample 0

The application should display “Array size must be larger than 0 message. Again, run gcov.


% gcov sample.c

84.62 % of 26 source lines executed in sample.c

Now comes the interesting part of testing this program. There are two malloc() error conditions; both must be tested to get 100% coverage of this code. Let’s use the gdb debugger to simulate the malloc() failures. Let’s set a break point and then jump to the error condition.


% gdb ./sample

The list command displays the line numbers for the source.


(gdb) list
12 int **array;
13 if (argc != 2) {

Use the break command to set a break point on line number 13.


(gdb) break 13

Then start the program with run.


(gdb) run
(gdb) list 30

36 if (array[x] == NULL)
37 printf(“Failed malloc for array size %d \n”, arraysize);

(gdb) jump 31

Malloc failed for array size

(gdb) quit


Once again, run gcov.


% gcov sample.c
92.31 % of 26 source lines executed in file sample.c

One more test to run. Follow the steps shown above to set a break point on line 13. Run the program with run and then jump to line 37.


% gdb ./sample
(gdb) break 13
(gdb) run

(gdb) list 30

36 if (array[x] == NULL)
37 printf(“Failed malloc for array size %d \n”, arraysize);

(gdb) jump 37
Failed malloc for array size

(gdb) quit


Finally, run gcov one last time.


% gcov sample.c

100 % of 26 source lines executed in file sample.c

Listing Three shows no lines flagged with #####, so all lines of this program have been executed. The number before each line of code tells how many times it was executed.




Listing One: sample.c, a test program

#include <stdlib.h>
#include <stdio.h>

int main(argc, argv)
int argc;
char **argv;
{
int x, y;
int arraysize;
int **array;
if (argc != 2) {
printf(“Usage: %s Enter arraysize value \n”,argv[0]);
exit(-1);
} else {
arraysize = atoi (argv[1]);
if (arraysize <= 0) {
printf(“Array size must be larger than 0 \n”);
exit(-1);
}
}

array = (int **) malloc (arraysize*sizeof (int *));

printf(“Creating an %d by %d array \n”, arraysize, arraysize);

if (array == NULL) {
printf(“Malloc failed for array size %d \n”, arraysize);
exit(-1);
}
for (x=0; x < arraysize; x++) {
array[x] = (int *) malloc (arraysize*sizeof (int));
if (array[x] == NULL) {
printf(“Failed malloc for array size %d \n”, arraysize);
exit(-1);
}
}
exit(0);
}




Listing Two: sample.c.gcov after running the application with input 1000


#include <stdlib.h>
#include <stdio.h>

int main(argc, argv)
int argc;
char **argv;
1 {
1 int x, y;
int arraysize;
int **array;
1 if (argc != 2) {
###### printf(“Usage: %s Enter arraysize value \n”,argv[0]);
###### exit(-1);

1 }
else {
1 arraysize = atoi (argv[1]);
1 if (arraysize <= 0) {
###### printf(“Array size must be larger than 0 \n”);
###### exit(-1);
1 }
1 }
1 array = (int **) malloc (arraysize*sizeof (int *));

1 printf(“Creating an %d by %d array \n”, arraysize, arraysize);

1 if (array == NULL) {
###### printf(“Malloc failed for array size %d \n”, arraysize);
###### exit(-1);
1 }
1001 for (x=0; x < arraysize; x++) {
1000 array[x] = (int *) malloc (arraysize*sizeof (int));
1000 if (array[x] == NULL) {
###### printf(“Failed malloc for array size %d \n”, arraysize);
###### exit(-1);
1000 }
1000 }
1 exit(0);
}




Listing Three: sample.c.gcov after the five tests

#include <stdlib.h>
#include <stdio.h>

int main(argc, argv)
int argc;
char **argv;
3 {
3 int x, y;
int arraysize;
int **array;
3 if (argc != 2) {
1 printf(“Usage: %s Enter arraysize value \n”,argv[0]);
1 exit(-1);

2 }
else {
2 arraysize = atoi (argv[1]);
2 if (arraysize <= 0) {
1 printf(“Array size must be larger than 0 \n”);
1 exit(-1);
1 }
1 }
1 array = (int **) malloc (arraysize*sizeof (int *));

1 printf(“Creating an %d by %d array \n”, arraysize, arraysize);

1 if (array == NULL) {
1 printf(“Malloc failed for array size %d \n”, arraysize);
1 exit(-1);
1 }
1001 for (x=0; x < arraysize; x++) {
1000 array[x] = (int *) malloc (arraysize*sizeof (int));
1000 if (array[x] == NULL) {
1 printf(“Failed malloc for array size %d \n”, arraysize);
1 exit(-1);
1000 }
1000 }
1 exit(0);
}

Coverage is Just One Measure

gcov determines how well your test suites exercise your code. One indirect benefit of gcov is that its output can be used to identify which test case provides coverage for each source file. With that information, you can run a subset of the test suite to validate changes in the program. Thorough code coverage during testing is one measurement of software quality.






lcov: The Linux Test Project’s gcov Extension


The Linux Test Project (LTP) is a suite of test cases for testing the Linux kernel. The goal of LTP is to deliver a suite of automated testing tools for Linux, as well as publish the results of the tests after they run. LTP uses gcov to determine the test coverage that its suites provide.

About a year ago, there was a patch produced to implement gcov support for the Linux kernel. After applying this patch and rebuilding the kernel, the standard gcov program can be used to see code coverage at the kernel level. For example, the attr.c file has 70.34% test coverage provided by the LTP test cases.

The LTP web page contains a link to work being done using their test suites. The kernel coverage map on the site provides coverage by kernel source code subdirectories. The summary also explains which test case provides coverage for each source code module in the kernel. This type of summary is helpful to determine which tests to run after source code has been changed in the kernel.





Resources


Documentation for gcov: http://gcc.gnu.org/onlinedocs/gcc/Gcov-Intro.html#Gcov%20Intro

The Linux Test Project (LTP): http://ltp.sourceforge.net

Linux Test Project GCOV extention (lcov) : http://ltp.sourceforge.net/lcov.php

Steve Best works in the Linux Technology Center of IBM in Austin, Texas. He is currently working on Journaled File System (JFS) for Linux project. Steve has done extensive work in operating system development with a focus in the areas of file systems, internationalization, and security. He can be reached at sbest@us.ibm.com. You can download the code used in this article from http://www.linux-mag.com/downloads/2003-07/compile.

Fatal error: Call to undefined function aa_author_bios() in /opt/apache/dms/b2b/linux-mag.com/site/www/htdocs/wp-content/themes/linuxmag/single.php on line 62