Skip navigation links
22.5.1.6 Using Server Logs to Find Causes of Errors in mysqld »
Section Navigation      [Toggle]

22.5.1.5. Using a Stack Trace

On some operating systems, the error log contains a stack trace if mysqld dies unexpectedly. You can use this to find out where (and maybe why) mysqld died. See Section 5.2.2, “The Error Log”. To get a stack trace, you must not compile mysqld with the -fomit-frame-pointer option to gcc. See Section 22.5.1.1, “Compiling MySQL for Debugging”.

A stack trace in the error log looks something like this: 

mysqld got signal 11;
Attempting backtrace. You can use the following information
to find out where mysqld died. If you see no messages after
this, something went terribly wrong...

stack_bottom = 0x41fd0110 thread_stack 0x40000
mysqld(my_print_stacktrace+0x32)[0x9da402]
mysqld(handle_segfault+0x28a)[0x6648e9]
/lib/libpthread.so.0[0x7f1a5af000f0]
/lib/libc.so.6(strcmp+0x2)[0x7f1a5a10f0f2]
mysqld(_Z21check_change_passwordP3THDPKcS2_Pcj+0x7c)[0x7412cb]
mysqld(_ZN16set_var_password5checkEP3THD+0xd0)[0x688354]
mysqld(_Z17sql_set_variablesP3THDP4ListI12set_var_baseE+0x68)[0x688494]
mysqld(_Z21mysql_execute_commandP3THD+0x41a0)[0x67a170]
mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x282)[0x67f0ad]
mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbb7[0x67fdf8]
mysqld(_Z10do_commandP3THD+0x24d)[0x6811b6]
mysqld(handle_one_connection+0x11c)[0x66e05e]

If resolution of function names for the trace fails, the trace contains less information: 

mysqld got signal 11;
Attempting backtrace. You can use the following information
to find out where mysqld died. If you see no messages after
this, something went terribly wrong...

stack_bottom = 0x41fd0110 thread_stack 0x40000
[0x9da402]
[0x6648e9]
[0x7f1a5af000f0]
[0x7f1a5a10f0f2]
[0x7412cb]
[0x688354]
[0x688494]
[0x67a170]
[0x67f0ad]
[0x67fdf8]
[0x6811b6]
[0x66e05e]

In the latter case, you can use the resolve_stack_dump utility to determine where mysqld died by using the following procedure:

  1. Copy the numbers from the stack trace to a file, for example mysqld.stack. The numbers should not include the surrounding square brackets: 

    0x9da402
0x6648e9
0x7f1a5af000f0
0x7f1a5a10f0f2
0x7412cb
0x688354
0x688494
0x67a170
0x67f0ad
0x67fdf8
0x6811b6
0x66e05e

  2. Make a symbol file for the mysqld server: 

    shell> nm -n libexec/mysqld > /tmp/mysqld.sym

    If mysqld is not linked statically, use the following command instead: 

    shell> nm -D -n libexec/mysqld > /tmp/mysqld.sym

    If you want to decode C++ symbols, use the --demangle, if available, to nm. If your version of nm does not have this option, you will need to use the c++filt command after the stack dump has been produced to demangle the C++ names.

  3. Execute the following command: 

    shell> resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack

    If you were not able to include demangled C++ names in your symbol file, process the resolve_stack_dump output using c++filt: 

    shell> resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack | c++filt

    This prints out where mysqld died. If that does not help you find out why mysqld died, you should create a bug report and include the output from the preceding command with the bug report.

    However, in most cases it does not help us to have just a stack trace to find the reason for the problem. To be able to locate the bug or provide a workaround, in most cases we need to know the statement that killed mysqld and preferably a test case so that we can repeat the problem! See Section 1.7, “How to Report Bugs or Problems”.


User Comments

Posted by Martin Mokrejs on January 12 2004 1:21am[Delete] [Edit]

This is a nice way to resolve the stack trace:

echo "This scripts expects stack trace from your mysql error log file"
echo "placed in the current directory as ./stack"

if [ ! -r ./stack ]; then
echo "./stack file not present or readable"
exit 255
fi

if [ -r /usr/local/mysql/bin/mysqld.sym.gz ]; then
gzip -dc /usr/local/mysql/bin/mysqld.sym.gz > ./symbols
else
echo "nm -n /usr/local/mysql/bin/mysqld > ./symbols"
nm -n /usr/local/mysql/bin/mysqld > ./symbols
fi

echo "/usr/local/mysql/bin/resolve_stack_dump -s ./symbols -n ./stack"
/usr/local/mysql/bin/resolve_stack_dump -s ./symbols -n ./stack

# if you have c++ filter, use rather this output, it's even better
echo "if you don't have c++filter, the next command will fail"
echo "/usr/local/mysql/bin/resolve_stack_dump -s ./symbols -n ./stack | c++filter"
/usr/local/mysql/bin/resolve_stack_dump -s ./symbols -n ./stack | c++filter

Posted by Martin Mokrejs on January 12 2004 1:53am[Delete] [Edit]

In my previous comment in the shellscript should be c++filt instead of c++filter. c++filt is a part of gcc package.

Add your own comment.