因 MTU 不一致导致主机和 RAC 数据库不断重启的问题排查与解决

JiekeXu之路 2020-03-23

4059

作者 | JiekeXu

来源 | JiekeXu之路（ID: JiekeXu_IT）

转载请联系授权 | (微信ID：xxq1426321293)

大家好，我是 JiekuXu,可以叫我老徐，也可称我强哥一个称呼怎么样都行，很高兴和大家又见面了。本文首发于微信公众号【JiekeXu之路】，欢迎点击上方蓝字关注我吧！

AIX 操作系统因 MTU 不一致导致主机和 RAC 数据库不断重启，事件就是发生在上周日。操作系统工程师因监控发现有一台主机不断重启，排查硬件后无问题，便将事件转至数据库工程师排查了。当时我接到的这个主机是一套不太重要的 SVC 存储复制的灾备环境，基本上没有业务使用，所以也就比较从容，慢慢排查问题。RAC 数据库是 11.2.0.4 版本的，操作系统是 AIX 7.1 版本，如下图：

MOS 上说该问题发生在 10.1.0.2 to 11.2.0.2 版本，但我这个库是 11.2.0.4，故记录一下排查过程。

Unable To Start ASM RAC Instances Due To ORA-27303: Remote Port MTU Does Not Match Local MTU. (Doc ID 947223.1)
Oracle Database - Enterprise Edition - Version 10.1.0.2 to 11.2.0.2 [Release 10.1 to 11.2]

MTU

MTU 是英文 Maximum Transmission Unit 的缩写，意为"最大传输单位"，也就是在连接的时候，所传输信息包最多可以有多少字节。我们必须找到不会返回 fragment (碎片)信息的最大 MTU。除了 ADSL PPPoE 的 MTU 是 1492 外，其余各种 DSL 的 MTU标准设置都是 1500。MaxMTU 是最大的 TCP/IP 传输单元，在 TCP/IP 协议中，将要传输的数据分成较小的组进行传输，每个组的大小为 576 字节。

AIX 系统错误日志存放路径：/var/adm/ras/errlog 。

errpt –a 列详细信息，详细使用方法可以参考 man。

AIX 系统启动错误日志存放路径：/var/adm/ras/bootlog。

该日志可以跟踪系统在 Boot 过程中发生的问题，包括服务器液晶板上的代码信息都有记载。可以使用 alog 命令监视这些问题, 存放在 var/adm/ras/bootlog 中，可以使用 alog –o –t boot 命令查看该文件。

问题排查

RAC 的节点二是出现问题的机器，主机每隔五六分钟就会重启，而节点一数据库启动后过几分钟也就宕了，这就十分奇怪了，主机二不断宕机会导致实例一宕机吗？故让我很好奇不断查下去了，以下是节点一的 alert 日志。

Sun Mar 15 19:06:11 2020
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.XX.12)(PORT=1521))' SCOPE=MEMORY SID='test1';
ALTER DATABASE MOUNT * db agent *//* {0:11:33} */
This instance was first to mount
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/test/control02.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/test/control02.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/test/control01.ctl'
ORA-17503: ksfdopn:2 Failed to open file +DATA/test/control01.ctl
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-205 signalled during: ALTER DATABASE MOUNT * db agent *//* {0:11:33} */...
Sun Mar 15 19:06:12 2020
Shutting down instance (abort)

以下是节点二 alert 日志，

Thu Mar 12 23:33:16 2020
NOTE: ASMB terminating
Errors in file app/oracle/diag/rdbms/test/test2/trace/test2_asmb_31457440.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 2 Serial number: 1
Errors in file app/oracle/diag/rdbms/test/test2/trace/test2_asmb_31457440.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 2 Serial number: 1
ASMB (ospid: 31457440): terminating the instance due to error 15064
Instance terminated by ASMB, pid = 31457440
Sun Mar 15 19:51:34 2020
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0

哦这里说一句，怎么查 alert 日志有很多小伙伴们不熟悉数据库便不知道日志位置，那么怎么查呢，登陆数据库查看 dump 参数指定的位置就是告警日志咯！

节点 1 ASM 日志，ORA-27303：additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)。ORA-27301、ORA-27302、ORA-27303 这里看到主要问题了吧。继续看一下节点二日志。

GMON started with pid=19, OS id=19857586 
Sun Mar 15 18:18:10 2020
MMON started with pid=20, OS id=19398838 
Sun Mar 15 18:18:10 2020
MMNL started with pid=21, OS id=24838354 
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 10)
ASM instance 
List of instances:
 1 2 (myinst: 1) 
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE 
KSXP IPC protocol is incompatible with instance 2
Errors in file app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_24707194.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)
KSXP IPC protocol is incompatible with instance 2
Errors in file app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_24707194.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 1500, remote: 9000] (169.254.1XX.XX8)
 Communication channels reestablished

节点 2 ASM 日志，ASM 日志存放于 ASM 实例之下，RAC 除了这两种日志外还有集群日志以及集群下的各种进程日志等多种，这个有时间在说。ASM 日志位于 $ORACLE_BASE/diag/asm/+asm/+ASM2/trace 下。

cd $ORACLE_BASE/diag/asm/+asm/+ASM2/trace
tail -999f app/grid/diag/asm/+asm/+ASM2/trace/alert*
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
ORACLE_HOME = app/product/11.2.0/grid
System name:    AIX
Node name:      test2
Release:        1
Version:        7
Machine:        00C0A6D84C00
Using parameter settings in server-side spfile +OCR/test-cluster/asmparameterfile/registry.253.1004572351
System parameters with non-default values:
  large_pool_size          = 12M
  instance_type            = "asm"
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskgroups           = "OCR"
  asm_diskgroups           = "ARCH"
  asm_diskgroups           = "DATA"
  asm_power_limit          = 1
  diagnostic_dest          = "/app/grid"
Cluster communication is configured to use the following interface(s) for this instance
  169.254.1XX.XX8
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Sun Mar 15 19:00:30 2020
PMON started with pid=2, OS id=22216894 
Sun Mar 15 19:00:30 2020
PSP0 started with pid=3, OS id=30539926 
Sun Mar 15 19:00:31 2020
VKTM started with pid=4, OS id=28573718 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Sun Mar 15 19:00:31 2020
GEN0 started with pid=5, OS id=22282482 
Sun Mar 15 19:00:31 2020
DIAG started with pid=6, OS id=29360284 
Sun Mar 15 19:00:31 2020
PING started with pid=7, OS id=21889258 
Sun Mar 15 19:00:31 2020
DIA0 started with pid=8, OS id=21823656 
Sun Mar 15 19:00:31 2020
LMON started with pid=9, OS id=28180628 
Sun Mar 15 19:00:31 2020
LMD0 started with pid=10, OS id=29819036 
* System load used for high load check 
* New Low - High Load Threshold Range = [55296 - 73728] 
Sun Mar 15 19:00:31 2020
LMS0 started with pid=11, OS id=30998728 at elevated priority
Sun Mar 15 19:00:31 2020
LMHB started with pid=12, OS id=27328766 
Sun Mar 15 19:00:31 2020
MMAN started with pid=13, OS id=22741166 
Sun Mar 15 19:00:31 2020
DBW0 started with pid=14, OS id=21495954 
Sun Mar 15 19:00:31 2020
LGWR started with pid=15, OS id=21168264 
Sun Mar 15 19:00:31 2020
CKPT started with pid=16, OS id=21954804 
Sun Mar 15 19:00:31 2020
SMON started with pid=17, OS id=30081278 
Sun Mar 15 19:00:31 2020
RBAL started with pid=18, OS id=27787438 
Sun Mar 15 19:00:31 2020
GMON started with pid=19, OS id=29425830 
Sun Mar 15 19:00:31 2020
MMON started with pid=20, OS id=27721860 
Sun Mar 15 19:00:31 2020
MMNL started with pid=21, OS id=30146734 
lmon registered with NM - instance number 2 (internal mem no 1)
Reconfiguration started (old inc 0, new inc 4)
ASM instance 
List of instances:
 1 2 (myinst: 2) 
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE 
KSXP IPC protocol is incompatible with instance 1
Errors in file app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_28180628.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 9000, remote: 1500] (169.254.1XX.XX)
KSXP IPC protocol is incompatible with instance 1
Errors in file app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_28180628.trc:
ORA-27300: OS system dependent operation:config_check failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvalpid
ORA-27303: additional information: Remote port MTU does not match local MTU. [local: 9000, remote: 1500] (169.254.1XX.XX)
 Communication channels reestablished

根据这些便可发现和 MOS （Doc ID 947223.1）所说一致了，MTU 不一致导致主机重启了。

当查看主机二的 MTU 时便惊奇的发现居然是 9000 ，而节点一均是 1500.这个就需要修改了，MTU 参数修改需要重启才会永久生效。所以需要把 en2 和 en3 改成 1500 才行。其他类型的系统查看如下：

On AIX:


# lsattr -El <adapter>
Example:


[celcaix4]/usr/sbin> lsattr -El en5
alias4 IPv4 Alias including Subnet Mask True
alias6 IPv6 Alias including Prefix Length True
arp on Address Resolution Protocol (ARP) True
authority Authorized Users True
broadcast Broadcast Address True
mtu 1500 Maximum IP Packet Size for This Device True <(== HERE
netaddr 10.xxx.xxx.91 Internet Address True
netaddr6 IPv6 Internet Address True
netmask 255.xxx.xxx.0 Subnet Mask True
prefixlen Prefix Length for IPv6 Internet Address True
remmtu 576 Maximum IP Packet Size for REMOTE Networks True
rfc1323 Enable/Disable TCP RFC 1323 Window Scaling True
security none Security Level True
state up Current Interface Status True
tcp_mssdflt Set TCP Maximum Segment Size True
tcp_nodelay Enable/Disable TCP_NODELAY Option True
tcp_recvspace Set Socket Buffer Space for Receiving True
tcp_sendspace Set Socket Buffer Space for Sending True




On Linux:


[root@asmstorage ~]# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:13:xx:xx:xx:46
inet addr:10.xxx.xxx.151 Bcast:10.xxx.xxx.255 Mask:255.xxx.xxx.0
inet6 addr: fe80::213:xxxx:xxxx:a146/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 <(== HERE
RX packets:8357638 errors:0 dropped:0 overruns:0 frame:0
TX packets:166322 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:776382023 (740.4 MiB) TX bytes:42281847 (40.3 MiB)
Interrupt:169




On Solaris:


# ifconfig -a
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
inet 127.xxx.xxx.1 netmask ff000000
eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index <(== HERE


inet 10.xxx.xxx.149 netmask fffff800 broadcast 10.xxx.xxx.255
ether 0:3:xx:xx:xx:d5