Oracle 11.2.0.4 RAC环境节点重新添加

DBA巫师 2024-04-08

361

点击上方蓝字关注我们

扫描下方二维码或添加作者微信，回复“加群”即可开启你的数据库和IT学习之旅。加入我们，你将获得不仅仅是知识，多位业内数据库ACE大佬镇场+免费GPT4资源+Oracle MOS免费查询，很多志同道合的小伙伴，欢迎加群一起探讨、一起学习、一起进步！

1.现有环境：

①数据库版本：oracle 11gR2 11.2.0.4；

②系统版本 Centos7；

③两个节点分别为ograc1 和 ograc2 （hostname与SID同名），RAC名称为ograc

④sys密码为 oracle

2.故障模拟：

我们遇到了一个问题，ograc2节点的磁盘突然出现了故障。需要注意的是，这块磁盘是我们用来安装和存放grid+oracle软件的u01目录。系统盘并未受到此次磁盘故障的影响。我们只更换了这块出现故障的磁盘，并成功地重新挂载了u01目录。

另一方面，ograc1节点的运行情况是正常的，没有发生任何故障或问题。

因ograc2节点的u01文件夹为新建文件夹，所以需要再次创建oracle及grid安装文件目录并赋权

mkdir -p u01/app/grid

mkdir -p /u01/app/11.2.0/grid

mkdir -p /u01/app/oraInventory

mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1

chown -R oracle:oinstall /u01/app/oracle

chown -R grid:oinstall /u01/app/grid

chown -R grid:oinstall /u01/app/oraInventory

chown -R grid:oinstall /u01/app/11.2.0/grid

chown -R root:oinstall /u01/software

chmod -R 775 /u01

3.修复过程

①在ograc1 节点下检查grid集群状态

[grid@ograc1 ~]$ crs_stat -t

[grid@ograc1 ~]$ crsctl stat res -t

在oracle用户下查看一下数据库情况

[oracle@ograc1 ~]$ srvctl config database -d ograc

确认一下数据库总共有两个节点，ograc1和ograc2,但根据之前的检查结果确定ograc2实例down掉

②此时确定ograc2节点掉线，迅速备份OCR

[root@ograc1 ~]# ocrconfig -export /tmp/ocr.20240218

③备份安装列表

[root@ograc1 ~]# cat /etc/oraInst.loc

[root@ograc1 ~]# cd /u01/app/oraInventory

[root@ograc1 oraInventory]# ll

[root@ograc1 oraInventory]# cd ContentsXML/

[root@ograc1 ContentsXML]# cp /u01/app/oraInventory/ContentsXML/inventory.xml /tmp/inventory.xml.bak

④ 开始修复

dbca 静默方式删除ograc2实例 (ograc1节点执行）

[oracle@ograc1 ~]$ dbca -silent -deleteInstance -nodeList ograc2 -gdbName ograc -instanceName ograc2 -sysDBAUserName sys -sysDBAPassword oracle

查看ograc.log

[oracle@ograc1 ~]$ cat /u01/app/oracle/cfgtoollogs/dbca/ograc.log

The user "oracle" does not have user equivalence setup on nodes "[ograc2]". Please set up the user equivalence before proceeding.

PRKC-1030 : Error checking accessibility for node ograc2, PRKC-1037 : Error removing files listed in ograc2 from node ssh: connect to host ograc2 port 22: No route to host

The Database Configuration Assistant will delete the Oracle instance and its associated OFA directory structure. All information about this instance will be deleted.

Do you want to proceed?

Deleting instance

Node ograc2 is not accessible. DBCA may not be able to remove all files for the instance. Do you want to continue to delete instance ograc2 on this node?

DBCA_PROGRESS : 1%

Unable to copy the file "ograc2:/etc/oratab" to "/tmp/oratab.ograc2".

DBCA_PROGRESS : 2%

DBCA_PROGRESS : 6%

DBCA_PROGRESS : 13%

DBCA_PROGRESS : 20%

DBCA_PROGRESS : 26%

DBCA_PROGRESS : 33%

DBCA_PROGRESS : 40%

DBCA_PROGRESS : 46%

DBCA_PROGRESS : 53%

DBCA_PROGRESS : 60%

DBCA_PROGRESS : 66%

Completing instance management.

DBCA_PROGRESS : 100%

·---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----

删除失效的rac节点

su - grid

[grid@ograc1 utl]$ cd /u01/app/11.2.0/grid/crs/utl

[grid@ograc1 utl]$ ./rootdeletenode.sh ograc2

执行后检查 crsctl stat res -t

·---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----

更新正常节点的安装列表，把节点信息更新为正常节点（ograc1）

su - oracle

[oracle@ograc1 ~]$ cd $ORACLE_HOME/oui/bin

[oracle@ograc1 bin]$ pwd

/u01/app/oracle/product/11.2.0/dbhome_1/oui/bin

[oracle@ograc1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 "CLUSTER_NODES={ograc1}"

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10239 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

su -grid

[grid@ograc1 grid]$ cd $ORACLE_HOME/oui/bin

[grid@ograc1 bin]$ pwd

/u01/app/11.2.0/grid/oui/bin

[grid@ograc1 bin]$ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={ograc1}" CRS=TRUE

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 10239 MB Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /u01/app/oraInventory

'UpdateNodeList' was successful.

·---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----

清理异常节点的CRS资源

su - root

[root@ograc1 ~]# srvctl stop vip -i ograc2

PRCC-1016 : ograc2-vip was already stopped

PRCR-1005 : Resource ora.ograc2.vip is already stopped

[root@ograc1 ~]# srvctl remove vip -i ograc2 -f

[root@ograc1 ~]# crsctl delete node -n ograc2

CRS-4661: Node ograc2 successfully deleted.

su - grid

[grid@ograc1 ~]$ cluvfy stage -post nodedel -n ograc2

Performing post-checks for node removal

Checking CRS integrity...

Clusterware version consistency passed

CRS integrity check passed

Node removal check passed

Post-check for node removal was successful.

[grid@ograc1 ~]$ cluvfy stage -post hwos -n ograc2

Performing post-checks for hardware and operating system setup

Checking node reachability...

Node reachability check passed from node "ograc1"

Checking user equivalence...

User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Check: Node connectivity for interface "ens33"

Node connectivity passed for interface "ens33"

TCP connectivity check passed for subnet "10.0.50.0"

Check: Node connectivity for interface "ens37"

Node connectivity passed for interface "ens37"

ERROR:

PRVF-7617 : Node connectivity between "ograc1 : 10.0.50.211" and "ograc2 : 192.168.144.212" failed

TCP connectivity check failed for subnet "192.168.144.0"

Node connectivity check failed

Checking multicast communication...

Checking subnet "10.0.50.0" for multicast communication with multicast group "230.0.1.0"...

Check of subnet "10.0.50.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.144.0" for multicast communication with multicast group "230.0.1.0"...

Check of subnet "192.168.144.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Check for multiple users with UID value 0 passed

Time zone consistency check passed

Checking shared storage accessibility...