暂无图片
误操作导致mysql无法启动的救援
最近更新:2022-02-21 15:02:04

起因:

一个mysql研发测试环境,单机,版本mysql5.7.24。

一线人员发现mysql挂了,磁盘占用100%. 于是他删除了部分binlog,腾出2G空间,启动mysql,正常。

因为不知道哪些数据表的数据可以truncate,他打算回收一些大表delete后的空间。找到一个最大表占14G的,optimize table table_name;发现空间一直上涨,因为会生成临时文件#sql34433.sql。于是想移走几个表腾出空间让optimize完成再移回来(错误操作,这时候没有停止mysql,应用还在读写)。发现mv几个表ibd文件后,空间没有腾出,此时optimize导致磁盘再次占满,mysql中断。

一线人员执行了lsof | grep deleted | awk ‘{ print $2 }’ | xargs -i kill -9 {};。回收了刚才mv的空间。重启mysql,无法启动。查看错误日志

2021-10-12 11:25:00 0x2b633c1c6700 InnoDB: Assertion failure in thread 47705210251008 in file btr0pcur.cc line 454

InnoDB: Failing assertion: btr_page_get_prev(next_page, mtr) == btr_pcur_get_block(cursor)->page.id.page_no()

InnoDB: We intentionally generate a memory trap.

InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

InnoDB: If you get repeated assertion failures or crashes, even

InnoDB: immediately after the mysqld startup, there may be

InnoDB: corruption in the InnoDB tablespace. Please refer to

InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html

......