rac开机后其中一个节点无法自动正常启动，请高手帮忙解答一下。

我来答

匿名用户

数据库版本：oracle 19.3 rac

操作系统：centos7.8

各种尝试后依然无法解决，请各位大神帮忙解决一下。

### 1.rac开机后其中一个节点无法自动正常启动

```bash
[grid@racdb1 ~]$ crsctl check cluster -all
**************************************************************
racdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racdb2:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
```

### 2.手动启动asm，可以启动。但是会报错OCR无法mount

```bash
SQL> startup
ASM instance started

Total System Global Area 1137173320 bytes
Fixed Size 8905544 bytes
Variable Size 1103101952 bytes
ASM Cache 25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "OCR" cannot be mounted
ORA-15040: diskgroup is incomplete
[grid@racdb2 trace]$ crsctl check cluster -all
**************************************************************
racdb1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racdb2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@racdb2 trace]$ srvctl status database -d orcl
Instance orcl1 is running on node racdb1
Instance orcl2 is running on node racdb2
```
### 3.修改asm_diskgroups仍然报错
查看asm_diskgroups参数发现修改asm磁盘组，创建spfile的时候没有删除原来的磁盘组OCR，这里直接修改asm_diskgroups。

```bash
SQL> show parameter asm_diskgroups;
VALUE
------------------------------
asm_diskgroups string
ARCH, DATA, OCRVOTE

SQL> alter system set asm_diskgroups='DATA','ARCH','OCRNEW'
```
再次重启节点依然无法开机自动启动，手动重启还是报刚才的错误。

### 4.查看日志
查看crs日志
/u01/app/grid/diag/crs/racdb2/crs/trace/alert.log
OCR 磁盘OCRVOTE无法mount
```bash
2022-09-14 09:49:31.017 [OHASD(2381)]CRS-2771: Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2022-09-14 09:52:48.515 [ORAROOTAGENT(2510)]CRS-5019: All OCR locations are on ASM disk groups [OCRVOTE], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/racdb2/crs/trace/ohasd_orarootagent_root.trc".
```
查看
/u01/app/grid/diag/crs/racdb2/crs/trace/ohasd_orarootagent_root.trc
这里看到关键错误：ORA-01017: invalid username/password
```bash
2022-09-14 09:44:38.140 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9348 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2022-09-14 09:44:38.148 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9348 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS

2022-09-14 09:44:38.217 : CLSCRED:4202641152: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.05101c2993facfe9bf726dfa23bbcd4b.root not found
2022-09-14 09:44:38.217 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] 9066 Error 4 opening dom root in 0x7fb0bc178b80

2022-09-14 09:44:38.275 : USRTHRD:4213147392: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2022-09-14 09:44:39.076 : USRTHRD:4213147392: [ INFO] {0:5:3} Thread:kgfoCheckMountExt Tasklet::doTask m_cv.timewait returned
2022-09-14 09:44:39.505 : default:4202641152: clsCredDomClose: Credctx deleted 0x7fb0bc255470
2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] -- trace dump on error exit --

2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3169

2022-09-14 09:44:39.505 : USRTHRD:4202641152: [ INFO] {0:5:3} [ora.storage] ORA-01017: invalid username/password; logon denied
```
再次重启节点依然无法开机自动启动，手动重启还是报刚才的错误。
### 5.修改asm密码文件，依然报错

[grid@racdb1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE TRUE
ASMSNMP TRUE FALSE FALSE
CRSUSER__ASM_001 TRUE FALSE TRUE

将CRSUSER__ASM_001改为CRSUSER__ASM_002也是一样结果

我来答

添加附件

问题补充

7条回答

默认

最新

刘贵宾

检查检查文件权限

有用 2

愤怒的蜗牛

2022-09-15

学习

许

[grid@racdb2 ~]$ ll /dev/asm*
brw-rw---- 1 grid asmadmin 253, 7 Sep 15 09:17 /dev/asm-disk3
brw-rw---- 1 grid asmadmin 253, 5 Sep 15 09:15 /dev/asm-disk6
brw-rw---- 1 grid asmadmin 253, 6 Sep 15 09:15 /dev/asm-disk7

主要是两问题

1.手工启动其中一个节点asm实例，为什么会报ORC无法mount

ORC已经被卸载

2.无法自动重启

报权限错误的问题

有用 0

Root__Liu

参考下：How to recreate shared ASM password file in 12c GI cluster (Doc ID 1929673.1)

有用 0

许

2022-09-15

[grid@racdb1 ~]$ asmcmd lspwusr Username sysdba sysoper sysasm SYS TRUE TRUE TRUE ASMSNMP TRUE FALSE FALSE CRSUSER__ASM_001 TRUE FALSE TRUE 将CRSUSER__ASM_001改为CRSUSER__ASM_002也是一样结果按照那些操作过。

刘贵宾

你在故障节点执行下这两个命令截图看一下

crsctl stat res -t

crsctl stat res -t -init

有用 1

许

2022-09-15

后面评论发出来了。

cqiwen

asm磁盘的权限感觉不对呀～把你的udev配置文件发出来看看呢

有用 0

许

2022-09-15

[grid@racdb2 ~]$ cat /etc/udev/rules.d/99-oracle-asmdevices.rules #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552384f546f46792d666c476c2d4e574d39", RUN+="/bin/sh -c 'mknod /dev/asm-disk2 b $major $minor; chown grid:asmadmin /dev/asm-disk2; chmod 0660 /dev/asm-disk2'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45526c7a6c46784d2d427633512d79636866", RUN+="/bin/sh -c 'mknod /dev/asm-disk3 b $major $minor; chown grid:asmadmin /dev/asm-disk3; chmod 0660 /dev/asm-disk3'" #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455277653366667a2d676576792d6a574934", RUN+="/bin/sh -c 'mknod /dev/asm-disk4 b $major $minor; chown grid:asmadmin /dev/asm-disk4; chmod 0660 /dev/asm-disk4'" #KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c45523741436b64482d3175424c2d54725944", RUN+="/bin/sh -c 'mknod /dev/asm-disk5 b $major $minor; chown grid:asmadmin /dev/asm-disk5; chmod 0660 /dev/asm-disk5'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c455239387a4954382d507758342d74304553", RUN+="/bin/sh -c 'mknod /dev/asm-disk6 b $major $minor; chown grid:asmadmin /dev/asm-disk6; chmod 0660 /dev/asm-disk6'" KERNEL=="dm-*", ENV{DEVTYPE}=="disk", SUBSYSTEM=="block", PROGRAM=="/usr/lib/udev/scsi_id -g -u -d $devnode", RESULT=="14f504e46494c4552447a5637447a2d4b5563352d554c7064", RUN+="/bin/sh -c 'mknod /dev/asm-disk7 b $major $minor; chown grid:asmadmin /dev/asm-disk7; chmod 0660 /dev/asm-disk7'" [grid@racdb2 ~]$ ls -l /dev/asm* brw-rw---- 1 grid asmadmin 253, 3 Sep 15 12:11 /dev/asm-disk3 brw-rw---- 1 grid asmadmin 253, 7 Sep 15 11:32 /dev/asm-disk6 brw-rw---- 1 grid asmadmin 253, 4 Sep 15 11:32 /dev/asm-disk7

许

重启节点后结果

[grid@racdb2 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racdb2 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racdb2 STABLE
ora.crf
1 ONLINE ONLINE racdb2 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE racdb2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racdb2 STABLE
ora.ctssd
1 ONLINE ONLINE racdb2 ACTIVE:0,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE racdb2 STABLE
ora.gipcd
1 ONLINE ONLINE racdb2 STABLE
ora.gpnpd
1 ONLINE ONLINE racdb2 STABLE
ora.mdnsd
1 ONLINE ONLINE racdb2 STABLE
ora.storage
1 ONLINE ONLINE racdb2 STABLE
--------------------------------------------------------------------------------
[grid@racdb2 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[grid@racdb2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Sep 15 12:10:20 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

Connected to an idle instance.

SQL>

开机后手动startup asm实列后的结果

[grid@racdb1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE racdb1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE racdb1 STABLE
ora.crf
1 ONLINE ONLINE racdb1 STABLE
ora.crsd
1 ONLINE ONLINE racdb1 STABLE
ora.cssd
1 ONLINE ONLINE racdb1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE racdb1 STABLE
ora.ctssd
1 ONLINE ONLINE racdb1 ACTIVE:0,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE racdb1 STABLE
ora.gipcd
1 ONLINE ONLINE racdb1 STABLE
ora.gpnpd
1 ONLINE ONLINE racdb1 STABLE
ora.mdnsd
1 ONLINE ONLINE racdb1 STABLE
ora.storage
1 ONLINE ONLINE racdb1 STABLE
--------------------------------------------------------------------------------
[grid@racdb1 ~]$
[grid@racdb1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.chad
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.net1.network
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
ora.ons
ONLINE ONLINE racdb1 STABLE
ONLINE ONLINE racdb2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ARCH.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE racdb1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE racdb1 Started,STABLE
2 ONLINE ONLINE racdb2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE racdb1 STABLE
2 ONLINE ONLINE racdb2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE racdb1 STABLE
ora.orcl.db
1 ONLINE ONLINE racdb1 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
2 ONLINE ONLINE racdb2 Open,HOME=/u01/app/o
racle/product/19.3.0
/db_1,STABLE
ora.qosmserver
1 ONLINE ONLINE racdb1 STABLE
ora.racdb1.vip
1 ONLINE ONLINE racdb1 STABLE
ora.racdb2.vip
1 ONLINE ONLINE racdb2 STABLE
ora.scan1.vip
1 ONLINE ONLINE racdb1 STABLE
--------------------------------------------------------------------------------

有用 0

许

问题未解决有没有人看一下。

有用 0

回答交流

提交

问题信息

请登录之后查看

邀请回答

暂无人订阅该标签，敬请期待~~

墨值悬赏