How do I debug an embedded Linux program using GDB?

Software development for embedded Linux systems is commonly done on a build host using a gcc cross compiler.  These compilers usually have support for the same debugging features that are used in native Linux development such as adding debugging information to the generated executables.  In native Linux, the ‘gdb’ command can be used to run the program in an environment where that debugging information can be put to use.  The most common use case for this is to get a stack trace when a crash happens, to help pinpoint the cause of a crash.  The ‘gdb’ command is not always available on embedded systems, but there is still a way to get stack traces using the ‘gdbserver’ command.  This command is more likely to be available on an embedded device when ‘gdb’ is not.  This is because gdbserver offloads most of the debugging work to an external computer, so it is suitable for less powerful processors.

The ‘gdbserver’ command allows remote debugging.  It runs on the target device and attaches to the process to be debugged.  It can be configured to create a TCP socket that the an external machine can connect to.  The following examples show two ways to start ‘gdbserver’ on a target device:

Attach to an already running process

Start a new process and attach to it

The external machine will need a version of gdb that supports the target architecture.  On many Linux platforms, this requires installing the ‘gdb-multiarch’ package.  Running the ‘gdb-multiarch’ command opens a gdb session that works like any other, but can also debug other supported architectures.  The following shows an example of starting a remote gdb session on the external machine:

Start a remote debugging session

After running the ‘target remote’ command, gdb will download all of the necessary binaries from the target machine and start a remote debugging session.