进程追踪与审计，SIGKILL分析，寻找进程被kill源头

一、前言

有时候我们在运维的时候会遇到一些奇怪的问题，某些进程突然不见了，无法得知是谁进行了kill的动作，我们改如何进行分析？

或者有时候我们通过strace分析到进行是被sigkill的，也就是kill -9；strace只能看出进程本身被kill了，无法看出是谁发起的，下面我们利用linux本身的audit工具进行审计分析，审计的手段比较多，这里实践演示audit。

二、实践步骤

1、工具安装

查看
[root@rac1 tmp]# rpm -qa |grep audit
audit-libs-2.8.5-4.el7.x86_64
audit-2.8.5-4.el7.x86_64
audit-libs-python-2.8.5-4.el7.x86_64

Note：如没有则配置yum进行安装

安装
yum install -y audit

2、服务启动

linux6：

service auditd start 
service auditd status

linux7：

systemctl start auditd
systemctl status auditd

[root@rac1 tmp]# systemctl status auditd
● auditd.service - Security Auditing Service
   Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2022-08-26 14:39:23 CST; 3 weeks 0 days ago
     Docs: man:auditd(8)
           https://github.com/linux-audit/audit-documentation
  Process: 5306 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
  Process: 5285 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
 Main PID: 5299 (auditd)
    Tasks: 5
   CGroup: /system.slice/auditd.service
           ├─5299 /sbin/auditd
           ├─5301 /sbin/audispd
           └─5303 /usr/sbin/sedispatch

Sep 13 18:42:45 rac1 auditd[5299]: Audit daemon rotating log files
Sep 14 03:31:12 rac1 auditd[5299]: Audit daemon rotating log files
Sep 14 12:19:50 rac1 auditd[5299]: Audit daemon rotating log files
Sep 14 21:01:30 rac1 auditd[5299]: Audit daemon rotating log files
Sep 15 05:43:56 rac1 auditd[5299]: Audit daemon rotating log files
Sep 15 14:36:27 rac1 auditd[5299]: Audit daemon rotating log files
Sep 15 23:19:00 rac1 auditd[5299]: Audit daemon rotating log files
Sep 16 07:56:03 rac1 auditd[5299]: Audit daemon rotating log files
Sep 16 15:41:58 rac1 auditd[5299]: Audit daemon rotating log files
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

3、审计规则添加

添加
auditctl -a exit,always -F arch=b64 -S kill -F key=catch_kill

Note：审计进程被kill的动作，添加key标记

查看
[root@rac1 tmp]# auditctl -l
-a always,exit -F arch=b64 -S kill -F key=catch_kill

4、模拟进程被KILL

这里我们随便发起一个top进程，等下将其kill掉

[root@rac1 ~]# ps -ef | grep top
oracle   10161 26279  0 15:28 pts/2    00:00:00 top
root     10320 26573  0 15:28 pts/3    00:00:00 grep --color=auto top

Note：这里我们看到top的进程的为10161，下面我们进行strace跟踪下被kill的动作

手动kill进程

[root@rac1 tmp]# kill -9 10161

查看strace结果

read(7, "0.26 0.42 0.51 1/1101 20333\n", 8191) = 28
lseek(4, 0, SEEK_SET)                   = 0
read(4, "cpu  40866110 1059 23151765 647118742 1415408 0 1133473 0 0 0\ncpu0 9980221 295 5"..., 2048) = 1486
read(4, "", 1024)                       = 0
write(1, "\33[H\33(B\33[mtop - 15:33:51 up 21 days, 54 min,  5 users,  load average: 0.26, 0.42,"..., 2048) = 2048
write(1, "4   4336   4128 S   1.3  0.1 365:11.03 asm_vktm_+asm1                           "..., 2048) = 2048
write(1, "2284264 227868 113800 S   0.3  3.8 221:25.02 ocssd.bin                          "..., 2048) = 2048
write(1, "0   0 1888588   4936   4368 S   0.3  0.1 126:12.61 ora_gcr0_oradb1              "..., 608) = 608
pselect6(1, [0], NULL, NULL, {3, 0}, {[WINCH], 8} <unfinished ...>
+++ killed by SIGKILL +++

Note：我们可以很清晰的看到最后一行进行被sigkill了

5、分析审计结果

查看结果
ausearch -sc kill

审计结果
time->Fri Sep 16 15:33:51 2022
type=PROCTITLE msg=audit(1663313631.744:1913209): proctitle="-bash"
type=OBJ_PID msg=audit(1663313631.744:1913209): opid=10161 oauid=0 ouid=11011 oses=7596 ocomm="top"
type=SYSCALL msg=audit(1663313631.744:1913209): arch=c000003e syscall=62 success=yes exit=0 a0=27b1 a1=9 a2=0 a3=7ffc911a6420 items=0 ppid=16272 pid=16280 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=7600 comm="bash" exe="/usr/bin/bash" key="catch_kill"
----

通过审计结果我们可以看到11011用户的10161的top进程被0号用户（root）的bash程序kill了；
那就可以知道是人为输入命令发起kill的动作。