Debugging "How To"

Getting core dump

A core dump is a special file which represents the memory image of a process. It is an important part of diagnosing the cause of the crash, since in the core dump is the data which the application was accessing at the time of crash, along with information about which part of the application was running at the time of crash.

To arrange to get core dump, at first use the following:

ulimit -c unlimited

to get no limit for core dumps. Then, if your program segfaults, you can use the debugger (gdb), which can read and interpret core dump files.

Example: debugging guile build error

# ulimit -c unlimited
# make V=1
...
make[3]: Entering directory `/root/build-farm/guile-v2.0.0-199-g0fbdbe6.builddir/libguile'
cat alist.doc arbiters.doc array-handle.doc array-map.doc arrays.doc async.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc expand.doc extensions.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-arrays.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc root.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc threads.doc throw.doc trees.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weaks.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc | GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools snarf-check-and-output-texi          > guile-procedures.texi || { rm guile-procedures.texi; false; }
/bin/bash: line 1: 18506 Broken pipe             cat alist.doc arbiters.doc array-handle.doc array-map.doc arrays.doc async.doc backtrace.doc boolean.doc bitvectors.doc bytevectors.doc chars.doc control.doc continuations.doc debug.doc deprecated.doc deprecation.doc dynl.doc dynwind.doc eq.doc error.doc eval.doc evalext.doc expand.doc extensions.doc feature.doc filesys.doc fluids.doc foreign.doc fports.doc gc-malloc.doc gc.doc gettext.doc generalized-arrays.doc generalized-vectors.doc goops.doc gsubr.doc guardians.doc hash.doc hashtab.doc hooks.doc i18n.doc init.doc ioext.doc keywords.doc list.doc load.doc macros.doc mallocs.doc memoize.doc modules.doc numbers.doc objprop.doc options.doc pairs.doc ports.doc print.doc procprop.doc procs.doc promises.doc r6rs-ports.doc random.doc rdelim.doc read.doc root.doc rw.doc scmsigs.doc script.doc simpos.doc smob.doc sort.doc srcprop.doc srfi-1.doc srfi-4.doc srfi-13.doc srfi-14.doc srfi-60.doc stackchk.doc stacks.doc stime.doc strings.doc strorder.doc strports.doc struct.doc symbols.doc threads.doc throw.doc trees.doc uniform.doc values.doc variable.doc vectors.doc version.doc vports.doc weaks.doc dynl.doc posix.doc net_db.doc socket.doc regex-posix.doc
     18507 Aborted                 (core dumped) | GUILE_AUTO_COMPILE=0 ../meta/uninstalled-env guile-tools snarf-check-and-output-texi > guile-procedures.texi
make[3]: *** [guile-procedures.texi] Error 1
...
# ls -t ./libguile/core* | head -1
./libguile/core.18507
# ./meta/gdb-uninstalled-guile
(gdb) core-file ./libguile/core.18507
[New Thread 18507]
Reading symbols from /root/build-farm/guile-v2.0.0-199-g0fbdbe6.builddir/libguile/.libs/libguile-2.0.so.22...done.
Loaded symbols for /root/build-farm/guile-v2.0.0-199-g0fbdbe6.builddir/libguile/.libs/libguile-2.0.so.22
Reading symbols from /usr/lib/libgc.so.1...done.
Loaded symbols for /usr/lib/libgc.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libffi.so.6...done.
Loaded symbols for /usr/lib/libffi.so.6
Reading symbols from /usr/lib/libunistring.so.0...done.
Loaded symbols for /usr/lib/libunistring.so.0
Reading symbols from /usr/lib/libgmp.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgmp.so.10
Reading symbols from /usr/lib/libltdl.so.7...done.
Loaded symbols for /usr/lib/libltdl.so.7
Reading symbols from /lib/libdl.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.0
Reading symbols from /lib/libcrypt.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.0
Reading symbols from /lib/libm.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.0
Reading symbols from /usr/lib/libgcc_s.so.1...done.
Loaded symbols for /usr/lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.0
Reading symbols from /lib/ld-uClibc.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-uClibc.so.0
Core was generated by `/root/build-farm/guile-v2.0.0-199-g0fbdbe6.builddir/libguile/.libs/lt-guile -e'.
Program terminated with signal 6, Aborted.
#0  0x48470744 in raise () from /lib/libc.so.0
(gdb) set logging on
Copying output to gdb.txt.
(gdb) set pagination off
(gdb) thread apply all bt
...
(gdb) quit
# cat gdb.txt

Thread 1 (Thread 18507):
#0  0x48470744 in raise () from /lib/libc.so.0
#1  0x4846aef0 in abort () from /lib/libc.so.0
#2  0x480ebc64 in scm_throw (key=0x0, args=0x10317ff0) at throw.c:102
#3  0x480e0f44 in scm_decoding_error (subr=<value optimized out>, err=38, message=<value optimized out>, port=<value optimized out>) at strings.c:1438
#4  0x480e1190 in scm_from_stringn (str=0x48128754 ??????, len=<value optimized out>, encoding=<value optimized out>, handler=<value optimized out>) at strings.c:1494
#5  0x480e11e8 in scm_from_locale_stringn (str=0x48128754 ??????, len=<value optimized out>) at strings.c:1535
#6  0x480e7304 in scm_from_locale_symboln (sym=<value optimized out>, len=<value optimized out>) at symbols.c:444
#7  0x480e7328 in scm_from_locale_symbol (sym=<value optimized out>) at symbols.c:438
#8  0x48050410 in scm_init_array_handle () at array-handle.c:138
#9  0x480894c0 in scm_i_init_guile (base=<value optimized out>) at init.c:399
#10 0x480e8e5c in scm_i_init_thread_for_guile (base=0xbfcd9fc8, parent=0x0) at threads.c:777
#11 scm_i_init_thread_for_guile (base=0xbfcd9fc8, parent=0x0) at threads.c:756
#12 0x480e8eb0 in with_guile_and_parent (base=0xbfcd9fc8, data=<value optimized out>) at threads.c:849
#13 0x4817f718 in GC_call_with_stack_base (fn=<value optimized out>, arg=<value optimized out>) at misc.c:1480
#14 0x480e9074 in scm_i_with_guile_and_parent (func=<value optimized out>, data=<value optimized out>) at threads.c:899
#15 scm_with_guile (func=<value optimized out>, data=<value optimized out>) at threads.c:905
#16 0x48089438 in scm_boot_guile (argc=<value optimized out>, argv=<value optimized out>, main_func=<value optimized out>, closure=<value optimized out>) at init.c:319
#17 0x100006c4 in main (argc=<value optimized out>, argv=<value optimized out>) at guile.c:70

Defining shortcuts, even persistent ones

The most commonly wanted information on each step is the contents of CPU registers. gdb does have "info registers" command to do that. But it's a great waste of time just to type it.

Great for us, gdb has define very well suitable to define shortcuts. Example:

Reading symbols from /usr/sbin/chroot...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/chroot . /usr/bin/screen
process 21539 is executing new program: /root/making_of_manulix/mx-chroot/usr/bin/screen-4.1.0
warning: no loadable sections found in added symbol-file /root/making_of_manulix/mx-chroot/usr/bin/screen-4.1.0

Program received signal SIGSEGV, Segmentation fault.
0xf7d2dd88 in malloc () from /root/making_of_manulix/mx-chroot/lib/libc.so.0
(gdb) define dr
Type commands for definition of "dr".
End with a line saying just "end".
>info registers
>end
(gdb) dr
r0             0xf7d2dcfc    4157791484
r1             0xfffbe890    4294699152
r2             0xf7e5e780    4159039360
r3             0x1    1
r4             0xf7cec6f0    4157523696
r5             0xf7d51fc0    4157939648
r6             0x10656000    275079168
r7             0x1    1
r8             0x1    1
r9             0x1000    4096
r10            0x1000    4096
r11            0xfffbe890    4294699152
r12            0x56    86
r13            0x10074698    268912280
r14            0x0    0
r15            0x0    0
r16            0x0    0
r17            0x0    0
r18            0x0    0
r19            0x0    0
r20            0x10657000    275083264
r21            0x28    40
r22            0xf7d57300    4157960960
r23            0x10657fff    275087359
r24            0x10656000    275079168
r25            0x1000    4096
r26            0xf7d57334    4157961012
r27            0x0    0
r28            0x1000    4096
r29            0xf7d57300    4157960960
r30            0xf7d51d84    4157939076
r31            0x18    24
pc             0xf7d2dd88    0xf7d2dd88 <malloc+1988>
msr            0xd032    53298
cr             0x20004042    536887362
lr             0xf7d2dcfc    0xf7d2dcfc <malloc+1848>
ctr            0x0    0
xer            0x0    0
orig_r3        0x10656000    275079168
trap           0x300    768

But this solution is one-session only. What if you do want to use dr as a shortcut for "info registers" every time you launch gdb? Well, it is simple! Just create "${HOME}"/.gdbinit file with the following inside it:

$ cat ~/.gdbinit
# dr - display registers
def dr
info registers
end

# dm - display memory
def dm
  if $argc >= 1
    x/16xw $arg0
  end
  if $argc == 0
    x/16xw $pc
  end
end

# il - instruction list
def il
  if $argc == 0
    x/8iw $pc
  end
  if $argc >= 1
    x/8iw $arg0
  end
end

And this shortcut would be persistent.

Unless otherwise stated, the content of this page is licensed under GNU Free Documentation License.