最近迁移恢复中遇到的几个小问题

在最近的2次迁移恢复中，遇到了如下几个错误，跟大家分享，并附上solution，小记一下~~~

###### 错误1

Sat Oct  6 05:15:48 2012
The input backup piece /backup/backup_control.bak is in compressed format.
Starting background process ASMB
ASMB started with pid=22, OS id=17282
Starting background process RBAL
RBAL started with pid=23, OS id=17286
Sat Oct  6 05:15:52 2012
Errors in file /u01/app/oracle/admin/ucp/bdump/ucp_rbal_17286.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Sat Oct  6 05:15:52 2012
Errors in file /u01/app/oracle/admin/ucp/bdump/ucp_rbal_17286.trc:
ORA-15183: ASMLIB initialization error [/opt/oracle/extapi/64/asm/orcl/1/libasm.so]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Sat Oct  6 05:15:52 2012
Errors in file /u01/app/oracle/admin/ucp/bdump/ucp_psp0_17246.trc:
ORA-00600: internal error code, arguments: [kfkLoadByNum03], [0x06000C4B0], [], [], [], [], [], []
Sat Oct  6 05:15:52 2012
Trace dumping is performing id=[cdmp_20121006051552]
Sat Oct  6 05:15:52 2012
Errors in file /u01/app/oracle/admin/ucp/bdump/ucp_psp0_17246.trc:
ORA-00600: internal error code, arguments: [kfkLoadByNum03], [0x06000C4B0], [], [], [], [], [], []
Sat Oct  6 05:15:52 2012
PSP0: terminating instance due to error 490
Sat Oct  6 05:15:52 2012
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/admin/ucp/bdump/ucp_diag_17244.trc
Sat Oct  6 05:15:52 2012
Trace dumping is performing id=[cdmp_20121006051552]
Sat Oct  6 05:15:53 2012
Instance terminated by PSP0, pid = 17246

oracle database 为10g，使用11.2.0.3的asm，进行恢复，即10g raw（rac）恢复到
单机的11.2.0.3 asm，报错。
最后将11gR2的asm磁盘直接dd清空掉，然后使用10g方式进行create diskgroup，然后启动asm实例，恢复成功。
此时就是使用11gR2 grid来管理10g asm，证明是可以的。


###### 错误2

channel ch03: reading from backup piece /backup/db_fullh0nn09qe_1568_1_20121005
ORA-19870: error reading backup piece /backup/db_fullgvnn09qc_1567_1_20121005
ORA-19504: failed to create file "+DATA/ucp/datafile/ucp_data_06.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/ucp/datafile/ucp_data_06.dbf
ORA-15055: unable to connect to ASM instance
ORA-00020: maximum number of processes () exceeded
ORA-15055: unable to connect to ASM instance
ORA-00020: maximum number of processes () exceede
ORA-19870: error reading backup piece /backup/db_fullh0nn09qe_1568_1_20121005
ORA-19504: failed to create file "+DATA/ucp/datafile/ucp_data_07.dbf"
ORA-17502: ksfdcre:4 Failed to create file +DATA/ucp/datafile/ucp_data_07.dbf
ORA-15055: unable to connect to ASM instance
ORA-00020: maximum number of processes () exceeded
ORA-15055: unable to connect to ASM instance
ORA-00020: maximum number of processes () exceede

asm实例参数processes过小，默认值为40，调整为200。

###### 错误3

客户在安装11gR2 rac（11.2.0.3.0）时，在第2节点执行root.sh报错，如下：

Disk Group CRS creation failed with the following message:
ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed


Configuration of ASM ... failed
see asmca logs at /u01/app/grid/cfgtoollogs/asmca for details
Did not succssfully configure and start ASM at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6763.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perlb -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed


其中 /u01/app/grid/cfgtoollogs/asmca 中的log信息如下：

[main] [ 2012-10-07 23:49:02.975 CST ] [UsmcaLogger.logInfo:143]  Instance running true
[main] [ 2012-10-07 23:49:03.011 CST ] [UsmcaLogger.logInfo:143]  Diskgroup does not exist, creating..
[main] [ 2012-10-07 23:49:03.042 CST ] [UsmcaLogger.logEntry:115]  Entering oracle.sysman.assistants.usmca.backend.USMDiskGroupManager Method : createDiskGroups
[main] [ 2012-10-07 23:49:03.043 CST ] [SQLEngine.doSQLSubstitution:2414]  The substituted sql statement:=select count(*) from v$ASM_DISKGROUP where name=upper('CRS')
[main] [ 2012-10-07 23:49:03.161 CST ] [UsmcaLogger.logInfo:143]  CREATE DISKGROUP SQL: CREATE DISKGROUP CRS NORMAL REDUNDANCY  DISK 'ORCL:OCRVDISK1', 
'ORCL:OCRVDISK2', 
'ORCL:OCRVDISK3' ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='1M'
[main] [ 2012-10-07 23:49:03.457 CST ] [SQLEngine.done:2189]  Done called
[main] [ 2012-10-07 23:49:03.460 CST ] [UsmcaLogger.logException:173]  SEVERE:method oracle.sysman.assistants.usmca.backend.USMDiskGroupManager:createDiskGroups
[main] [ 2012-10-07 23:49:03.460 CST ] [UsmcaLogger.logException:174]  ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed

[main] [ 2012-10-07 23:49:03.461 CST ] [UsmcaLogger.logException:175]  oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-15018: diskgroup cannot be created
ORA-15072: command requires at least 2 regular failure groups, discovered only 0
ORA-15080: synchronous I/O operation to a disk failed

oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1658)
oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeSql(SQLEngine.java:1925)
oracle.sysman.assistants.usmca.backend.USMDiskGroupManager.createDiskGroups(USMDiskGroupManager.java:239)
oracle.sysman.assistants.usmca.backend.USMDiskGroupManager.createDiskGroups(USMDiskGroupManager.java:124)
oracle.sysman.assistants.usmca.backend.USMDiskGroupManager.createDiskGroupsLocal(USMDiskGroupManager.java:2212)
oracle.sysman.assistants.usmca.backend.USMInstance.configureLocalASM(USMInstance.java:3069)
oracle.sysman.assistants.usmca.service.UsmcaService.configureLocalASM(UsmcaService.java:1049)
oracle.sysman.assistants.usmca.model.UsmcaModel.performConfigureLocalASM(UsmcaModel.java:944)
oracle.sysman.assistants.usmca.model.UsmcaModel.performOperation(UsmcaModel.java:797)
oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:174)
oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:369)
[main] [ 2012-10-07 23:49:03.462 CST ] [UsmcaLogger.logExit:124]  Exiting oracle.sysman.assistants.usmca.backend.USMDiskGroupManager Method : createDiskGroups
[main] [ 2012-10-07 23:49:03.463 CST ] [UsmcaLogger.logInfo:143]  Diskgroups created
[main] [ 2012-10-07 23:49:03.463 CST ] [UsmcaLogger.logInfo:143]  Diskgroup creation is not successful.
[main] [ 2012-10-07 23:49:03.464 CST ] [UsmcaLogger.logExit:124]  Exiting oracle.sysman.assistants.usmca.model.UsmcaModel Method : performConfigureLocalASM
[main] [ 2012-10-07 23:49:03.465 CST ] [UsmcaLogger.logExit:124]  Exiting oracle.sysman.assistants.usmca.model.UsmcaModel Method : performOperation

在oracle mos中查到2个文档，分别如下：

1）Root.Sh Script Fails On Second Node with ASMCA ORA-15018 [ID 1303409.1]）

and also we can see that +ASM1 is running on both nodes
 
[node1-+ASM1]/dbs/trace> ps -ef|grep pmon
 
oracle    3989  3811  0 00:49 pts/3    00:00:00 grep pmon
oracle   22626     1  0 Mar11 ?        00:00:00 asm_pmon_+ASM1
 

[node2-+ASM2]/dbs/trace> ps -ef|grep pmon
 
oracle   15670     1  0 Mar11 ?        00:00:00 asm_pmon_+ASM1
oracle   27337 27175  0 00:49 pts/2    00:00:00 grep pmon
 
The disks are not shared across the nodes.hence the second node also trying to perform the initial configuration and ASMCA is failing.

2） ORA-15072 when creating a diskgroup with external redundancy [ID 396015.1]

If EMC based storage but use the normal Linux multipath driver is used, then please make the following changes in the /etc/sysconfig/oracleasm file.

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER="multipath sd"

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan ORACLEASM_SCANEXCLUDE="sd"


最后实际上就是修改 /etc/sysconfig/oracleasm 文件中的如下2个参数解决了该问题：

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

需要注意的是，在rac的所有节点都需要进行修改，不然执行root.sh仍然会报错。

###### 错误4

在进行rman restore时，发现无法进行restore,明显是oracle bug，显示oracle并不为认为这是bug。 错误如下：

Reread of blocknum=1, file=/u01/app/oracle/UCP/oradata/undotbs1_04.dbf. found valid data
Mon Oct 08 03:59:53 2012
Full restore complete of datafile 28 to datafile copy /u01/app/oracle/UCP/oradata/web_index_04.dbf.  Elapsed time: 0:05:52 
  checkpoint is 24788316
  last deallocation scn is 3
Mon Oct 08 04:00:01 2012
Full restore complete of datafile 27 to datafile copy /u01/app/oracle/UCP/oradata/web_index_03.dbf.  Elapsed time: 0:06:36 
  checkpoint is 24788723
  last deallocation scn is 3
Mon Oct 08 04:02:07 2012
Full restore complete of datafile 26 to datafile copy /u01/app/oracle/UCP/oradata/web_index_02.dbf.  Elapsed time: 0:08:22 
  checkpoint is 24788119
  last deallocation scn is 3
Mon Oct 08 04:02:59 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8153.trc  (incident=36407):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Mon Oct 08 04:02:59 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8156.trc  (incident=36427):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:02:59 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8155.trc  (incident=36419):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Mon Oct 08 04:02:57 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8154.trc  (incident=36411):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8155.trc  (incident=36420):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8154.trc  (incident=36412):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8156.trc  (incident=36428):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8153.trc  (incident=36408):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8156 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8154 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8155 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8153 
Mon Oct 08 04:03:03 2012
Sweep [inc][36428]: completed
Sweep [inc][36427]: completed
Sweep [inc][36420]: completed
Sweep [inc][36419]: completed
Sweep [inc][36412]: completed
Sweep [inc][36411]: completed
Sweep [inc][36408]: completed
Sweep [inc][36407]: completed
Mon Oct 08 04:09:05 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8889.trc  (incident=36371):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:09:05 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8886.trc  (incident=36299):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:09:05 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8887.trc  (incident=36307):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:09:05 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8888.trc  (incident=36349):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8889.trc  (incident=36372):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8886.trc  (incident=36300):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8887.trc  (incident=36308):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8889 
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_8888.trc  (incident=36350):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8886 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8887 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 8888 
Mon Oct 08 04:09:06 2012
Sweep [inc][36372]: completed
Sweep [inc][36371]: completed
Sweep [inc][36350]: completed
Sweep [inc][36349]: completed
Sweep [inc][36308]: completed
Sweep [inc][36307]: completed
Sweep [inc][36300]: completed
Sweep [inc][36299]: completed
Mon Oct 08 04:13:18 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9397.trc  (incident=36351):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:13:18 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9399.trc  (incident=36373):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:13:18 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9396.trc  (incident=36309):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:13:18 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9398.trc  (incident=36363):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9397.trc  (incident=36352):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9399.trc  (incident=36374):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9396.trc  (incident=36310):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9398.trc  (incident=36364):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9397 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9399 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9396 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9398 
Mon Oct 08 04:13:21 2012
Sweep [inc][36374]: completed
Sweep [inc][36373]: completed
Sweep [inc][36364]: completed
Sweep [inc][36363]: completed
Sweep [inc][36352]: completed
Sweep [inc][36351]: completed
Sweep [inc][36310]: completed
Sweep [inc][36309]: completed
Mon Oct 08 04:19:26 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9903.trc  (incident=36375):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:19:26 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9900.trc  (incident=36353):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:19:26 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9899.trc  (incident=36311):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Oct 08 04:19:26 2012
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9902.trc  (incident=36365):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9903.trc  (incident=36376):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9900.trc  (incident=36354):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9899.trc  (incident=36312):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/ucp/UCP3/trace/UCP3_ora_9902.trc  (incident=36366):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9903 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9900 
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9899 
Mon Oct 08 04:19:27 2012
Sweep [inc][36376]: completed
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 9902 

在oracle mos中有如下一篇文档进行了描述：ORA-00600 [kfioTranslateIO03] [17090] [ID 1336846.1]

Cause
 group permission of "oracle" executable from RDBMS home should have the same group information for ASM devices according to note 1084186.1.

$ ls -l $GRID_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 228954465 Jul 1 13:37 /oh1/grid/product/11.2.0/bin/oracle

$ ls -l $RDBMS_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 228954465 Jul 1 13:37 /oh1/oracle/product/11.2.0/bin/oracle

$ ls -l $ASM_DEVICE/* 
brw-rw---- 1 grid asmadmin 8, 33 Feb 15 08:11 /dev/oracleasm/disks/ASMD1
brw-rw---- 1 grid asmadmin 8, 49 Feb 15 08:11 /dev/oracleasm/disks/ASMD2
brw-rw---- 1 grid asmadmin 8, 17 May 4 22:30 /dev/oracleasm/disks/CRSD1

But in this case, "oracle" executable from RDBMS shows different group information which is different from group information for ASM devices.
ORA-600[kfioTranslateIO03] and [17090] occurrs due to the permission issue. 

$ ls -l $RDBMS_HOME/bin/oracle
-rwsr-s--x 1 oracle oinstall 228954465 Jul 1 13:37 /oh1/oracle/product/11.2.0/bin/oracle
 ^^^^^^^ it should be "asmadmin" or at least should be the same group of all ASM devices.

 
Solution
 
 group information for $RDBMS_HOME/bin/oracle should be changed to the group that can read/write to ASM devices. 

Please execute the following action plan from note 1084186.1.

$ su - grid
$ cd /bin
$ ./setasmgidwrap o=<11.2 RDBMS Home>/bin/oracle

最后所有节点执行如下命令：

su - grid
cd $GRID_HOME/bin
./setasmgidwrap o=/u01/app/oracle/product/11.2.0/db_1/bin/oracle

注意，是所有节点，因为后面在其他2个节点启动实例时发现异常，执行后错误消失。