Session 9: Debugging Programs

The goal of debugging and performance optimization is to create code that runs as correctly and quickly as possible. Debugging is the process by which programming errors are found and corrected. Performance optimization is the analysis and improvement of the algorithms and methods implemented in the code.

Getting started

In this session, we will use examples in either C, C++ or Fortran90. Choose your preferred language of the three and download the files to be used in this session by either clicking one of the following three links: C version, C++ version, F90 version, or by copying the relevant files on ManeFrame with one of the following 3 commands:

$ cp /hpc/examples/workshops/hpc/session9_c.tgz .
$ cp /hpc/examples/workshops/hpc/session9_cxx.tgz .
$ cp /hpc/examples/workshops/hpc/session9_f90.tgz .

Unpack your tarball and enter the resulting directory.

Debugging and debuggers

Enabling Debugging Information

In most compilers (including GNU and PGI), you can enable debugging information through adding the -g compiler flag. Add this flag to the compilation commands in the Makefile for the target driver2.exe, and then compile the executable,

$ make driver2.exe

Run the new executable. It should die with an error message about a segmentation violation (segmentation fault) or bus error, depending on the compiler/OS, e.g.

$ ./driver2.exe
Segmentation fault

There are many ways to track down this kind of error (e.g. adding print statements everywhere, staring intently hoping for an epiphany, randomly changing things to see what happens). In this session we will use the most efficient debugging approach, that of using a tool to track down the bug for us.

The tool we will use is the GNU debugger, which can be accessed through running the faulty executable program from within the debugging program itself. Load the executable into gdb with the command

$ gdb driver2.exe

At the gdb prompt, type run to start the executable. It will automatically stop at the line where the segmentation fault occurs.

In another terminal window, you can type man gdb to learn more about how to use the debugger (or you can click here to view the gdb man page on the web.

  • Perhaps the most valuable gdb command is print that may be used to see the internal value of a specified variable, e.g.

    (gdb) print i

    will print out the current value of the iteration variable i).

  • The help command inside of gdb may be used to find out more information on how to use the program itself.

  • The quit command inside of gdb will exit the debugger and return you to the command line. Alternatively, you may just type ^d ([control]-[d]) to exit.

Fixing the Bug

C users:
Open both the files driver2.c and tridiag_matvec.c, and see if you can find/fix the problem by using gdb and print statements as appropriate.
C++ users:
Open both the files driver2.cpp and tridiag_matvec.cpp, and see if you can find/fix the problem by using gdb and print statements as appropriate.
F90 users:
Open both the files driver2.f90 and tridiag_matvec.f90, and see if you can find/fix the problem by using gdb and print statements as appropriate.

A word of warning, the location of the segmentation fault or bus error is not always where the problem is located. Segmentation faults generally occur due to an attempt within the program to read to or write from an illegal memory location, i.e. a memory location that is not a part of a currently-available variable. Examples of bugs that can cause a seg-fault are iterating outside of the bounds of an array, or a mismatch between the arguments that a program uses to call a function and the arguments that the function expects to receive.


Tips for tracking/fixing segmentation faults

Using a debugger:

  1. determine exactly the line of code causing the fault,

  2. if the fault is inside a loop, determine exactly which iteration of the loop is causing the fault,

  3. use print statements in the debugger to see which variable is uninitialized, e.g. to see if the array x has entry i you could use

    (gdb) print x[i]

Once you identify the precise location of the segmentation fault, go back to see where the data is allocated. Was it allocated with a different size, shape or type? Was it not allocated at all?

If the data is allocated in a different manner than it is being used, determine which location needs fixing and try your best.

Upon finding and fixing the bug causing the segmentation fault, the correctly-executing program should write the following line:

2-norm of product = 1.414213562373E+00

(or something within roundoff error of this result), and it should write the file r.txt that contains the result of the matrix-vector product. This output vector should contain all 0’s except for the first and last entries, which should be 1.

Advanced debuggers

There are many freely-available Linux debugging utilities in addition to gdb. Most of these are graphical (i.e. point-and-click), and in fact use gdb under the hood. Some of the more popular of these debuggers include: ddd, nemiver, eclipse, zerobugs, edb. However, of this set the ManeFrame cluster currently only has gdb installed (ask your system administrators for others you want/need).

Additionally, there are some highly advanced non-free Linux debugging utilities available (all typically graphical), including TotalView, DDT, idb (only works with the Intel compilers), and PGI’s pgdbg (graphical) and pgdebug (text version). Of these, the ManeFrame cluster has both pgdbg and pgdebug.

The usage of most of the above debuggers is similar to gdb, except that in graphical debuggers it can be easier to view the data/instruction stack. The primary benefit of the non-free debuggers is their support for debugging parallel jobs that use OpenMP, MPI, or hybrid MPI/OpenMP computing approaches (see session 9). In fact, some of these professional tools can even be used to debug code running on GPU accelerators.

If you’re interested in learning more about these, I recommend that you re-download the tarball for this session, load the pgi module, update the Makefile to use the -g option along with the relevant PGI compiler (pgcc, pgc++ or pgfortran), and launch the job in the pgdbg debugger like you did with gdb:

$ pgdbg ./driver2.exe

Press the “play” button to start the executable running, and use the mouse to interact with the debugger as needed.


SMU pays for a five-seat PGI license, meaning that only five distinct compilation/debugging processes with the PGI tools may be run simultaneously. Typically, five is much more than sufficient for a campus of our size, since users spend most of their time writing code, preparing input parameters and scripts for running simulations, or post-processing simulation data; the time spent actually compiling and using a debugger is minimal. However, if everyone in the workshop tries this simultaneously, we would obviously exceed the five “seats,” which is why this is left as a personal exercise.