万里数据库GreatDB集中式故障诊断：coredump处理流程-4

现场初步分析

分析error log

首先需要在error log中，搜索backtrace关键字查找异常堆栈。

67912 Build ID: 0eaf4b944b1dbc99a26f9343f301e033bdedeb1d
67913 Server Version: 8.0.25-15-greatdbcluster5.0.7-GA GreatDB Cluster, Release GA, Revision b43d2b2e462
67914
67915 Thread pointer: 0x7f914ec27000
67916 Attempting backtrace. You can use the following information to find out
67917 where mysqld died. If you see no messages after this, something went
67918 terribly wrong...
67919 stack_bottom = 7f91608d8c30 thread_stack 0x46000
67920 /greatdb/svr/greatdb/bin/greatdbd(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x200c33d]
67921 /greatdb/svr/greatdb/bin/greatdbd(handle_fatal_signal+0x37b) [0x1169e4b]
67922 /lib64/libpthread.so.0(+0xf5d0) [0x7f91d95c55d0]
67923 /greatdb/svr/greatdb/lib/plugin/ha_greatdb.so(greatdb::ha_greatdb::build_order_list_string(String&)+0x6c) [0x7f915951aa4c]
67924 /greatdb/svr/greatdb/lib/plugin/ha_greatdb.so(greatdb::ha_greatpart::rnd_init_low(bool)+0x1ac) [0x7f915959765c]
67925 /greatdb/svr/greatdb/bin/greatdbd(handler::ha_rnd_init(bool)+0x22) [0xcc6102]
67926 /greatdb/svr/greatdb/bin/greatdbd(TableScanIterator::Init()+0x4e) [0xeed66e]
67927 /greatdb/svr/greatdb/bin/greatdbd(LimitOffsetIterator::Init()+0x16) [0x122afe6]
67928 /greatdb/svr/greatdb/bin/greatdbd(Query_expression::ExecuteIteratorQuery(THD*)+0x281) [0x10a95a1]
67929 /greatdb/svr/greatdb/bin/greatdbd(Query_expression::execute(THD*)+0x2f) [0x10a994f]
67930 /greatdb/svr/greatdb/bin/greatdbd(Sql_cmd_dml::execute(THD*)+0x516) [0x1029376]
67931 /greatdb/svr/greatdb/bin/greatdbd(mysql_execute_command(THD*, bool)+0x9e0) [0xfcce70]
67932 /greatdb/svr/greatdb/bin/greatdbd(dispatch_sql_command(THD*, Parser_state*, bool)+0x4f1) [0xfd0961]
67933 /greatdb/svr/greatdb/bin/greatdbd(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1843) [0xfd26b3]
67934 /greatdb/svr/greatdb/bin/greatdbd(do_command(THD*)+0x210) [0xfd3c70]
67935 /greatdb/svr/greatdb/bin/greatdbd() [0x115ac90]
67936 /greatdb/svr/greatdb/bin/greatdbd() [0x24cf38e]
67937 /lib64/libpthread.so.0(+0x7dd5) [0x7f91d95bddd5]
67938 /lib64/libc.so.6(clone+0x6d) [0x7f91d796dead]
67939
67940 Trying to get some variables.
67941 Some pointers may be invalid and cause the dump to abort.
67942 Query (7f913ef65028): select * from `grid-report`.`dp_bi_main_order_acccum` limit 0, 1000
67943 Connection ID (thread ID): 53
67944 Status: NOT_KILLED

可以从这里看到如下几个信息：

1. Query (7f913ef65028): select * from grid-report.dp_bi_main_order_acccum limit 0, 1000语句导致的堆栈异常
2. 最后core在了build_order_list_string函数，且函数内的偏移是0x6c
3. Server Version: 8.0.25-15-greatdbcluster5.0.7-GA GreatDB Cluster, Release GA, Revision b43d2b2e462, 这里是commit版本号，我们需要使用发布对应这个版本号的二进制。