Chapter 5. Debugging MPI Applications

Debugging MPI applications can be more challenging than debugging sequential applications. This chapter presents methods for debugging MPI applications. It covers the following topics:

MPI Routine Argument Checking

By default, the SGI MPI implementation does not check the arguments to some performance-critical MPI routines such as most of the point-to-point and collective communication routines. You can force MPI to always check the input arguments to MPI functions by setting the MPI_CHECK_ARGS environment variable. However, setting this variable might result in some degradation in application performance, so it is not recommended that it be set except when debugging.

Using the TotalView Debugger with MPI programs

The syntax for running SGI MPI with the TotalView Debugger (TVD) from TotalView Technologies is, as follows:

% totalview mpirun -a -np 4 a.out

Note that TVD is not expected to operate with MPI processes started via the MPI_Comm_spawn or MPI_Comm_spawn_multiple functions.

The MPT mpiexec_mpt(1) command has a -tv option for use by MPT with the TotalView Debugger. Note that the PBS Professional mpiexec(1) command does not support the -tv option.

To run an MPT MPI job with TotalView without a batch scheduler (same as the above example), perform the following:

% totalview mpirun -a -np 4 a.out

To run an MPT MPI job with Total View Debugger with a batch schduler, such as, PBS Professional or Torque, perform the following:

% mpiexec_mpt -tv -np 4 a.out

Using idb and gdb with MPI programs

Because the idb and gdb debuggers are designed for sequential, non-parallel applications, they are generally not well suited for use in MPI program debugging and development. However, the use of the MPI_SLAVE_DEBUG_ATTACH environment variable makes these debuggers more usable.

If you set the MPI_SLAVE_DEBUG_ATTACH environment variable to a global rank number, the MPI process sleeps briefly in startup while you use idb or gdb to attach to the process. A message is printed to the screen, telling you how to use idb or gdb to attach to the process.

Similarly, if you want to debug the MPI daemon, setting MPI_DAEMON_DEBUG_ATTACH sleeps the daemon briefly while you attach to it.