Remote debugging of embedded systems

GDB, as a server, must be compiled into the debugging target build in order to support connected GDB clients. When running on the client side, there must exist a copy of the target source as well as an unstripped (of symbols) version of the executable.

GCC compilation should be done with the -g flag.

On the target/server side, run with
gdbserver host:<port> --attach <pid> or gdbserver host:<port> <program>
You can use a serial port device, like /dev/ttyS0 or localhost:<port>. The gdbserver does not need the symbol table, so it can be given a stripped version.

On the client side, run gdb <program,unstripped> and then
(gdb) target remote IP:PORT

What gdb client to run? Must be built with the right target processor in mind, for example
.../toolchain/bin/powerpc-linux-gdb core/mydaemon/src/mydaemon -x gdb-command-script
-x is a filename option. Usually it will come with the toolchain for the target. If not, you can build it from source. When you use the ./configure tool, set --target to be the embedded target you want to debug on. The --host flag is the gdb client connecting to the embedded target.

Note that you can load the target's executable from the host/client, you don't have to load it from the target.

What about running the debugger on the target?

Note that if you want to run gdb client on the target, you are looking for a different set of tools entirely. You'll want the so-called bare metal toolchain, the Debian package gdb-arm-none-eabi for example which is actually compiled for embedded processors. But the easiest way if you use Buildroot is to select it in the package options under 'Debugging, profiling and benchmark' and make sure you choose 'full debugger', otherwise the default is just to include the gdbserver.

.gdbinit File

It's handy to use a .gdbinit file to set up the connection command, and point to the target libraries.

Might look something like this:

set solib-absolute-prefix /home/Software/lib/rootfs.unstripped
directory /home/Software/lib
directory /home/Software/lib/libawesome/include
directory /home/Software/lib/libawesome/src
directory /home/Software/lib/libsweet/include
directory /home/Software/lib/libsweet/src

set listsize 30
set print vtbl on
set print object on
set print pretty on
set unwindonsignal on
target remote 1.1.1.1:1111

You can use GDB to generate kernel signals, and also intercept signals that the target triggers.

To enable the use of a .gdbinit file from any launch folder, create a ~/.gdbinit file and insert the line set auto-load safe-path .

What if I move the repo to another machine and GDB gets lost with an old path? Use

set substitute-path /old_path /new_path

Command files

A command sequence file can be specified on the command line invocation of gdb client, which might look like this

#this is a comment
break CoolObject:FunctionName
continue
tbreak OKObject:GetStuff
continue

To use, launch gdb with the -x option followed by the name of the command file.

File Compilation

Make sure you have the -g flag set for the compiler and that -s is NOT set for the linker, otherwise you'll get a stripped binary.

-s
       Omit all symbol information from the output file.
-S
       Omit debugger symbol information (but not all symbols) from the output file.

Command tips

Shortcuts

b = break
c = continue
s = step
p = print
n = next
l = list current code

step is generally "step into" while next is "step over".

Awesome GDB Uses

Use gdb to redirect output, say from a serial port to a file, of a running process. In theory you could do this with stdout and stderr.
https://stackoverflow.com/questions/1323956/how-to-redirect-output-of-an-already-running-process
Other methods to achieve this are relying on the output being sent to a virtual serial port, not a physical device. I have found no other way to achieve this besides using gdb.

Helpful commands:
https://ccrma.stanford.edu/~jos/stkintro/Useful_commands_gdb.html

Building gdb from source

For Embedded Target As Host

The maintainers use autotools for some awful reason. Really hard to get it to work cross compiled. I finally hit on the right combination:

LD=/opt/toolch7700/armv5-eabi--glibc--stable/bin/arm-linux-ld 
CC=/opt/toolch7700/armv5-eabi--glibc--stable/bin/arm-linux-gcc 
CXX=/opt/toolch7700/armv5-eabi--glibc--stable/bin/arm-linux-g++ 
AR=/opt/toolch7700/armv5-eabi--glibc--stable/bin/arm-linux-ar 
./configure --host=arm-linux-gnueabi
make

With host set to arm-linux it had problems, but the above worked!!

For Linux Workstation As Host

./configure --target=arm-linux-gnueabi --prefix=<installdir>

If you get this annoying error on every line after building your own gdb: Python Exception <type 'exceptions.NameError'> Installation error: gdb.execute_unwinders function is missing: then your installation is messed up. Make sure you set a --prefix for configure and after make do make install.

Problems

Even if you get this error Cannot find bounds of current function stepi should still work.

Kicking off with run using gdbserver will not work, and you'll get The "remote" target does not support "run".

To be sure everything runs correctly, you should make sure to build your executable with gcc debug option -g and optimizations cleared -O0, otherwise breakpoints may be missed and line numbers may be slightly skewed.