ogg异常故障排查之日志缺失

今天发现进程挂掉，查看report说是磁盘满，再看一下磁盘空间发现已经清理了，启动该进程，之后进程又挂掉了。
排查过程如下：

1、查看report报告

2023-12-22 11:59:00 INFO OGG-01851 filecaching started: thread ID: 140399863658240.

2023-12-22 11:59:00 INFO OGG-01815 Virtual Memory Facilities for: COM
anon alloc: mmap(MAP_ANON) anon free: munmap
file alloc: mmap(MAP_SHARED) file free: munmap
target directories:
/u01/ogg/dirtmp.

Database Version:
MySQL
Server Version: 5.7.17-log
Client Version: 5.6.14
Host Connection: Localhost via UNIX socket
Protocol Version: 10

Database Language and Character Set:
CLIENT CHARACTER SET = "utf8mb4"
SERVER CHARACTER SET = "utf8mb4"
DATABASE CHARACTER SET = "utf8mb4"
LOCALE INFORMATION = "en"
DATABASE COLLATION = "caseinsensitive"

2023-12-22 11:59:03 INFO OGG-01056 Recovery initialization completed for target file /u01/ogg/dirdat/exp/mo000000032, at RBA 134394099, CSN 000000000000000001785:00000
0559079734.

2023-12-22 11:59:03 INFO OGG-01478 Output file /u01/ogg/dirdat/exp/mo is using format RELEASE 12.3.

2023-12-22 11:59:03 INFO OGG-01026 Rolling over remote file /u01/ogg/dirdat/exp/mo000000032.

2023-12-22 11:59:03 INFO OGG-00182 VAM API running in single-threaded mode.

2023-12-22 11:59:03 INFO OGG-01513 Positioning to Log Number: 1785
Record Offset: 558921462.

INFO !! using index file /data/dbdata/binary-log.index

Source Context :
SourceModule : [ggvam.gen]
SourceID : [/scratch/aime/adestore/views/aime_adc4150325/oggcore/OpenSys/src/gglib/ggvam/cvamgen.cpp]
SourceMethod : [vamInitialize]
SourceLine : [543]
ThreadBacktrace : [14] elements
: [/lib64/libc.so.6(__libc_start_main)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract()]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract()]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**)]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*))]
: [/u01/ogg/extract(ggs::gglib::MultiThreading::MainThread::Run(int, char**))]
: [/u01/ogg/extract(main)]
: [/lib64/libc.so.6(__libc_start_main)]
: [/u01/ogg/extract()]

2023-12-22 11:59:04 ERROR OGG-00146 Call to VAMInitialize returned with error status 600: VAM Client Report <CAUSE OF FAILURE : Failed to retrieve log file name
WHEN FAILED : While setting initial position in the binary log file
WHERE FAILED : MySQLBinLog Reader Module
CONTEXT OF FAILURE : No Information Available!>.

2023-12-22 11:59:04 ERROR OGG-01668 PROCESS ABENDING.

根据日志报错信息，发现是日志缺失，但是没有显示是哪个日志

2、查看mysql日志

查看mysql保留的日志情况

mysql> show binary logs;
+-------------------+------------+
| Log_name          | File_size  |
+-------------------+------------+
| binary-log.001787 | 1079559914 |
| binary-log.001788 | 1082219273 |
| binary-log.001789 | 1074907793 |
| binary-log.001790 |  201447013 |
+-------------------+------------+

查看当前目录binlog日志

srwxrwxrwx 1 mysql mysql          0 Dec 14 17:08 mysql.sock
-rw-r----- 1 mysql mysql 1079559914 Dec 18 05:22 binary-log.001787
-rw-r----- 1 mysql mysql 1082219273 Dec 20 05:28 binary-log.001788
-rw-r----- 1 mysql mysql 1074907793 Dec 22 05:24 binary-log.001789
-rw-r----- 1 mysql mysql        124 Dec 22 05:24 binary-log.index

查看发现日志保留最早的是12.18号的日志。

3、abend进程

GGSCI (bejrun-n-22025 DBLOGIN as ogg) 8>                            info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     ABENDED     EXP_1       00:00:01      203:24:32   

4、处理

从进程挂起情况来看已经挂起了200多小时，而mysql的日志保留时间只有五天，所以进程挂起。找到问题原因好了，重新初始化进程，进程恢复正常