1，遇到错误信息！
删除1号节点，重新添加节点遇到报错！
hydata02[ / ]#/oracle/crs/product/10.2.0/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Node hydata01 is already assigned nodenum 3.
Aborting: No configuration data has been changed.
clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...
[-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]
-nn specifies nodenames in the same fashion as -nn in -install mode
-pn specifies private interconnect names as -pn in -install mode
-hn specifies hostnames in the same fashion as -hn in -install mode
-t specifies port numbers to be used by CRS daemons on the new node(s)
default ports: 49895,49896,49897,49898
WARNING: Using this tool may corrupt your cluster configuration. Do not
use unless you positively know what you are doing.
/oracle/crs/product/10.2.0/crs/bin/srvctl add nodeapps -n hydata01 -A hydata01_vip/255.255.255.0/lan900 -o /oracle/crs/product/10.2.0/crs
CRS-0210: Could not find resource 'ora.hydata01.vip'.
2，经查询ocr文件损坏导致
Repairing or Restoring an Inconsistent OCR in RAC (Doc ID 268937.1)
SCOPE & APPLICATION
-------------------
This article is intended for DBAs and Support Engineers who need to correct
an inconsistent OCR (Oracle Configuration Repository).
REPAIRING OR RESTORING AN INCONSISTENT OCR (ORACLE CONFIGURATION REPOSITORY)
----------------------------------------------------------------------------
If you have encountered a condition where you are unable to add or remove a CRS
resource, your OCR file may be inconsistent. Example error when removing a CRS
resource with SRVCTL:
PRKS-1028 : Configuration for <CRS resource> does not exist in cluster registry.
or
CRS-0210: Could not find resource
Example error when adding a resource with SRVCTL:
PRKS-1003 : Failed to register CRS resource...
POSSIBLE CAUSES OF OCR INCONSISTENCY
3，恢复OCR
#OCR备份信息查询$ORA_CRS_HOME/bin/ocrconfig -showbackup [root@ract1 rac_db1]# $ORA_CRS_HOME/bin/ocrconfig -showbackup ract1 2014/04/11 14:54:52 /u01/app/oracle/product/crs/cdata/crs10g ract1 2014/04/10 18:32:29 /u01/app/oracle/product/crs/cdata/crs10g ract1 2014/04/10 12:55:12 /u01/app/oracle/product/crs/cdata/crs10g ract1 2014/04/10 12:55:12 /u01/app/oracle/product/crs/cdata/crs10g ract2 2014/04/07 08:06:05 /u01/app/oracle/product/crs/cdata/crs10g恢复OCR文件：/sbin/init.d/init.crs disable
/sbin/init.d/init.crs stop
cd /oracle/app/oracle/product/10.2.0/db_1/bin/
ls -l /oracle/crs/product/10.2.0/crs/cdata/crs/week.ocr
# . /home/oracle/.profile（root 用户需要声明环境变量信息，否则命令不能使用！！！）
./ocrconfig -restore /oracle/crs/product/10.2.0/crs/cdata/crs/week.ocr
/sbin/init.d/init.crs enabe
/sbin/init.d/init.crs start
# 参考信息
hydata02[ / ]#/oracle/crs/product/10.2.0/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Node hydata01 is already assigned nodenum 3.
Aborting: No configuration data has been changed.
clscfg -add -nn nameA,numA,nameB,numB,... -pn privA,numA,privB,numB,...
[-hn hostA,numA,hostB,numB,...] [-t p1,p2,p3,p4]
-nn specifies nodenames in the same fashion as -nn in -install mode
-pn specifies private interconnect names as -pn in -install mode
-hn specifies hostnames in the same fashion as -hn in -install mode
-t specifies port numbers to be used by CRS daemons on the new node(s)
default ports: 49895,49896,49897,49898
WARNING: Using this tool may corrupt your cluster configuration. Do not
use unless you positively know what you are doing.
/oracle/crs/product/10.2.0/crs/bin/srvctl add nodeapps -n hydata01 -A hydata01_vip/255.255.255.0/lan900 -o /oracle/crs/product/10.2.0/crs
CRS-0210: Could not find resource 'ora.hydata01.vip'.
CRS-210 error and OCR corruption
https://www.hhutzler.de/blog/restore-ocr-after-crs-210-error-and-ocr-corruption/
$ORA_CRS_HOME/bin/ocrconfig -showbackup
https://www.cnblogs.com/l10n/p/9416816.html
https://blog.csdn.net/loryliu/article/details/47978317
https://www.cnblogs.com/l10n/p/9416816.html
#CRS-210 error and OCR corruption – Error: could not find ora.ract3.vip
https://www.hhutzler.de/blog/restore-ocr-after-crs-210-error-and-ocr-corruption/
Repairing or Restoring an Inconsistent OCR in RAC (Doc ID 268937.1)
Repairing or Restoring an Inconsistent OCR in RAC (Doc ID 268937.1) To BottomTo Bottom
PURPOSE
-------
To provide a method of repairing or restoring an inconsistent OCR in 10.1 through 11.1.
SCOPE & APPLICATION
-------------------
This article is intended for DBAs and Support Engineers who need to correct
an inconsistent OCR (Oracle Configuration Repository).
REPAIRING OR RESTORING AN INCONSISTENT OCR (ORACLE CONFIGURATION REPOSITORY)
----------------------------------------------------------------------------
If you have encountered a condition where you are unable to add or remove a CRS
resource, your OCR file may be inconsistent. Example error when removing a CRS
resource with SRVCTL:
PRKS-1028 : Configuration for <CRS resource> does not exist in cluster registry.
or
CRS-0210: Could not find resource
Example error when adding a resource with SRVCTL:
PRKS-1003 : Failed to register CRS resource...
POSSIBLE CAUSES OF OCR INCONSISTENCY
------------------------------------
1. The "-f" option of srvctl. Do not use the -f (force) option in srvctl, the
-f flag causes it to ignore any errors and will remove whatever pieces it can
anyway. If you don't use "-f" on the remove command, then srvctl would stop when
it encountered an error, and would not remove the OCR entries. You can run
srvctl remove without the -f as many times as you need without causing an
inconsistency. The only time a "-f" option should be used is if you have
verified that you have an inconsistency in the OCR and you are trying to
correct it.
2. CRS_Unregister was used when it was not needed. You should use srvctl remove
to remove CRS resources. CRS_Unregister should only be used if you are trying
to repair an inconsistency.
3. The rootdeletenode.sbs script has a "-f" srvctl command in it. This is a
known issue. Best practice is to modify rootdeletenode.sbs to not do a "-f".
Bug 3566785 - ROOTDELETENODE SHOULD NOT HAVE -F FLAG ON REMOVE NODEAPPS
[This section is not visible to customers.]
4. Another potential cause of inconsistency is running multiple commands to
configure the same object at the same time. Like multiple "srvctl add service"
commands to add different services to the same database at once.
Bug 3068488 - PERFORMANCE: CAN'T ADD SERVICES IN PARALLEL
[This section is not visible to customers.]
FILES TO GATHER FOR OCR INCONSISTENCY
-------------------------------------
If you have a reproducible testcase that creates inconsistency in the OCR,
please open a service request and provide the exact steps to Oracle Support
Services. The following are some files that may need to be reviewed for issues
involving OCR inconsistencies:
- CRS Log files:
10.1:
cd $ORA_CRS_HOME
tar cf /var/backup/crs.tar crs/init crs/log css/init css/log evm/init evm/log srvm/log
10.2:
cd $ORA_CRS_HOME
tar cf log
or run diagcollection.pl
- OCR dump file - To get this cd to $ORA_CRS_HOME/bin as the root user and issue
"ocrdump". This will generate two files (ocrdump.log and OCRDUMPFILE).
- $ORA_CRS_HOME/bin/crs_stat -u and $ORA_CRS_HOME/bin/crs_stat -p output
- srvctl config output:
srvctl config database -d <db_name> -a
srvctl config service -d <db_name> -a
srvctl config nodeapps -n <node name>
srvctl config nodeapps -n <node name> -a -g -s -l
srvctl config asm -n <node name>
- Trace output of SRVCTL commands. Set the environment variable SRVM_TRACE to
true prior to running srvctl commands.
FIXING INCONSISTENT OCR FILES
-----------------------------
The following methods can be used to correct an inconsistent OCR file:
Method 1: Repair the OCR
Method 2: Restore the OCR from a backup.
Method 3: Re-install CRS
METHOD 1 - REPAIR THE OCR
-------------------------
If you cannot use srvctl to remove a CRS resource, find out if there is
information missing in CRS or in SRVM. To do this, run
$ORA_CRS_HOME/bin/crs_stat to see if the CRS resource exists. Also use
one of the following srvctl commands to see if the resource exists in
SRVM:
srvctl config database -d <db_name> -a
srvctl config service -d <db_name> -a
srvctl config nodeapps -n <node name>
srvctl config nodeapps -n <node name> -a -g -s -l
srvctl config asm -n <node name>
You may either see information missing from the srvctl config command
or from the crs_stat command. If you are unable to find an
inconsistency and you cannot remove the resource with srvctl, open a
service request or proceed to method 2.
SRVCTL CONFIG DATA IS MISSING BUT CRS_STAT INFORMATION IS PRESENT
-----------------------------------------------------------------
If information is missing from srvctl config, you can attempt to use
crs_unregister. You should only use crs_unregister commands with the
understanding that your OCR may need to be restored anyway. If
crs_unregister does not work you may need to either restore or re-create
your OCR. To use crs_unregister:
1. Get the resource name of the resource you are trying to remove with
$ORA_CRS_HOME/bin/crs_stat. Example:
cd $ORA_CRS_HOME/bin
./crs_stat
You will see CRS resources, example of a CRS resource:
NAME=ora.V10SN.V10SN2.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on
2. Attempt to unregister the CRS resource with crs_unregister. Example:
cd $ORA_CRS_HOME/bin
./crs_unregister ora.V10SN.V10SN2.inst
3. If your original goal was to add a CRS resource, try to use srvctl to add
the resource. If your original goal was to remove a CRS resource, verify
that it is removed from crs_stat and srvctl config.
If there is still a problem with the OCR, proceed to method 2.
CRS_STAT DATA IS MISSING BUT SRVCTL CONFIG INFORMATION IS PRESENT
-----------------------------------------------------------------
If information is missing from crs_stat, you can attempt to use
srvctl remove -f. You should only use srvctl remove -f commands with the
understanding that your OCR may need to be restored anyway. If
srvctl remove -f does not work you may need to either restore or re-create
your OCR. To use srvctl remove -f:
1. Determine what resource exists in srvctl config but is missing from crs_stat
2. Remove the resource with the -f option in srvctl.
srvctl remove database -d <database-name> -f
srvctl remove instance -d <database-name> [-i <instance-name>] -f
srvctl remove service -d <database-name> -s <service-name> [-i <instance-name>] -f
srvctl remove nodeapps -n <node-name> -f
3. If your original goal was to add a CRS resource, try to use srvctl to add
the resource. If your original goal was to remove a CRS resource, verify
that it is removed from crs_stat and srvctl config.
METHOD 2 - RESTORE THE OCR FROM A BACKUP
----------------------------------------
Note: See the documentation for your CRS version for updated/exact steps.
If crs_unregister or srvctl remove -f does not fix the OCR problem, you may
need to restore the OCR from a backup. Oracle automatically takes backups of
the OCR every 4 hours. Oracle also keeps the last 3 backups, up to 4 hours old,
one day old, and one week old available. Here are the steps for restoring the
OCR.
1. Find out what time the problem that the inconsistency in the OCR occurred.
2. Find an OCR backup from a time prior to when the inconsistency occurred.
To do this run $ORA_CRS_HOME/bin/ocrconfig -showbackup. The output mentions
the hostname on which the backup is located and the timestamp. Otherwise
check <CRS_HOME>/cdata/<cluster name> (the default dir) on all nodes for
backups.
Note that you can also run 'ocrdump -stdout -backupfile <backup file name>'
to check the contents of the backup to help find the most appropriate one.
If a backup OCR cannot be located. It may also be possible to use OCR
export/import if an export is available. See the Oracle documentation
for detailed steps on how to do this.
3. If you have a backup of the OCR from prior to the time of the inconsistency,
reboot the nodes in single user mode or runlevel 1. If you are unable to
reboot into single user mode for some reason, you can disable CRS with:
Sun or Linux:
/etc/init.d/init.crs disable
/etc/init.d/init.crs stop
HP-UX or HP Tru64::
/sbin/init.d/init.crs disable
/sbin/init.d/init.crs stop
IBM AIX:
/etc/init.crs disable
/etc/init.crs stop
4. After all nodes are rebooted in single user mode and/or you have verified that
CRS is not running (ps -ef | grep crs), restore the OCR with ocrconfig.
Example:
cd $ORA_CRS_HOME/bin
./ocrconfig -restore /t02/app/oracle/product/crs/cdata/crs_opcbsol/week.ocr
5. Re-enable CRS if it was disabled. Example:
Sun or Linux:
/etc/init.d/init.crs enable
HP-UX or HP Tru64::
/sbin/init.d/init.crs enable
IBM AIX:
/etc/init.crs enable
6. Reboot the nodes or re-start the clusterware.
METHOD 3 - RE-CREATE OCR AND VOTING DISK(S)
-------------------------------------------
The OCR and voting disk files can be recreated with the instructions provided
in the following article:
Note:399482.1 - How to recreate OCR/Voting disk accidentally deleted
METHOD 4 - RE-INSTALL CRS
-------------------------
If all else fails, re-install CRS. Only do this after consulting with Oracle
Support Services and there is no reasonable way to fix the inconsistency.
1. Use Note 239998.1 to completely remove the CRS installation.
2. Re-install CRS
3. Run the CRS root.sh as prompted at the end of the CRS install.
4. Run the root.sh in the database $ORACLE_HOME to re-run VIPCA. This will re-
create the VIP, GSD, and ONS resources.
5. Use NETCA to re-add any listeners.
6. Add databases and instances with SRVCTL, syntax is in Note 259301.1
RELATED DOCUMENTS
-----------------
Note 259301.1 CRS and 10g Real Application Clusters
Note 178683.1 - Tracing GSD, SRVCTL, GSDCTL, and SRVCONFIG
Note 239998.1 - 10g RAC How to Clean Up After a Failed CRS Install
