CRS-5802: Unable to start the agent process

racdb资源状态显示offline，使用crs_start启动数据库报错如下：
[code]ora.crm.db
1 OFFLINE OFFLINE crmdb1
2 OFFLINE OFFLINE crmdb2

root@crmdb1[/oracle/grid/app/product/11.2.0/gi/bin] # ./crs_start ora.crm.db
Attempting to start ora.crm.db on member crmdb2
Attempting to start ora.crm.db on member crmdb1
Start of ora.crm.db on member crmdb2 failed.
Attempting to stop ora.crm.db on member crmdb2
Start of ora.crm.db on member crmdb1 failed.
Attempting to stop ora.crm.db on member crmdb1
ora.crm.db on member crmdb2 has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process
ora.crm.db on member crmdb1 has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process
CRS-0215: Could not start resource 'ora.crm.db 1 1'.

CRS-0215: Could not start resource 'ora.crm.db 2 1'.

ora.hxcrm.db
1 ONLINE UNKNOWN crmdb1
2 ONLINE UNKNOWN crmdb2[/code]

报错无法启动agent程序，查看crsalert日志和crsd日志：
[code]root@hxcrmdb1[/oracle/grid/app/product/11.2.0/gi/log/crmdb1] # vi alertcrmdb1.log

2014-03-16 12:03:49.615:
[crsd(5373982)]CRS-5828:Could not start agent '/oracle/grid/app/product/11.2.0/gi/bin/oraagent_oracle'. Details at (:C
RSAGF00130:) {1:42956:1285} in /oracle/grid/app/product/11.2.0/gi/log/hxcrmdb1/crsd/crsd.log.

root@hxcrmdb1[/oracle/grid/app/product/11.2.0/gi/log/hxcrmdb1/crsd] # vi crsd.log

2014-03-16 01:12:28.891: [ CRSPE][11310] {1:42956:422} CRS-2679: Attempting to clean 'ora.hxcrm.db' on 'hxcrmdb2'

2014-03-16 01:12:28.892: [ AGFW][10025] {1:42956:422} Created alert : (:CRSAGF00130:) : Failed to start the agent
/oracle/grid/app/product/11.2.0/gi/bin/oraagent_oracle
2014-03-16 01:12:28.893: [ AGFW][10025] {1:42956:422} Agfw Proxy Server sending the last reply to PE for message:RE
SOURCE_START[ora.hxcrm.db 1 1] ID 4098:2178
2014-03-16 01:12:28.894: [ AGFW][10025] {1:42956:422} Can not stop the agent: /oracle/grid/app/product/11.2.0/gi/bi
n/oraagent_oracle because pid is not initialized
2014-03-16 01:12:28.894: [ CRSPE][11310] {1:42956:422} Received reply to action [Start] message ID: 2178
2014-03-16 01:12:28.894: [ CRSPE][11310] {1:42956:422} RI [ora.hxcrm.db 1 1] new internal state: [STABLE] old value:
[STARTING]
2014-03-16 01:12:28.894: [ CRSPE][11310] {1:42956:422} Fatal Error from AGFW Proxy: Unable to start the agent proces
s
2014-03-16 01:12:28.894: [UiServer][11567] {1:42956:422} Container [ Name: ORDER
MESSAGE:
TextMessage[CRS-2679: Attempting to clean 'ora.hxcrm.db' on 'hxcrmdb2']
MSGTYPE:
TextMessage[3]
OBJID:
TextMessage[ora.hxcrm.db 2 1]
WAIT:
TextMessage[0]
][/code]
查看agent进程：
[code]oracle@hxcrmdb2:/oracle$ps -ef |grep agent
grid 5177442 1 0 23:33:17 - 1:32 /oracle/grid/app/product/11.2.0/gi/bin/oraagent.bin
grid 5374064 1 0 23:33:57 - 0:10 /oracle/grid/app/product/11.2.0/gi/bin/scriptagent.bin
root 2622076 2557246 0 Mar 14 - 1:13 /var/opt/tivoli/ep/_jvm/jre/bin/java -Xmx384m -Xminf0.01 -Xmaxf0.4 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Xbootclasspath/a:/var/opt/tivoli/ep/runtime/core/eclipse/plugins/com.ibm.rcp.base_6.2.3.20110824-0615/rcpbootcp.jar:/var/opt/tivoli/ep/lib/com.ibm.logging.icl_1.1.1.jar:/var/opt/tivoli/ep/lib/jaas2zos.jar:/var/opt/tivoli/ep/lib/jaasmodule.jar:/var/opt/tivoli/ep/lib/lwidiag.jar:/var/opt/tivoli/ep/lib/lwinative.jar:/var/opt/tivoli/ep/lib/lwinl.jar:/var/opt/tivoli/ep/lib/lwirolemap.jar:/var/opt/tivoli/ep/lib/lwisecurity.jar:/var/opt/tivoli/ep/lib/lwitools.jar:/var/opt/tivoli/ep/lib/passutils.jar:../../runtime/agent/lib/cas-bootcp.jar -Xverify:none -cp eclipse/launch.jar:eclipse/startup.jar:/var/opt/tivoli/ep/runtime/core/eclipse/plugins/com.ibm.rcp.base_6.2.3.20110824-0615/launcher.jar com.ibm.lwi.LaunchLWI
grid 3998244 1 0 23:32:14 - 0:48 /oracle/grid/app/product/11.2.0/gi/bin/oraagent.bin
oracle 3932972 3671500 0 13:24:49 pts/1 0:00 grep agent
root 4391694 1 1 23:33:14 - 10:46 /oracle/grid/app/product/11.2.0/gi/bin/orarootagent.bin
root 1901586 1704422 0 Mar 14 - 0:00 /opt/freeware/cimom/pegasus/bin/cimssys platform_agent
root 4326576 1 0 23:32:18 - 0:30 /oracle/grid/app/product/11.2.0/gi/bin/cssdagent
root 4654266 1 0 23:32:13 - 2:33 /oracle/grid/app/product/11.2.0/gi/bin/orarootagent.bin
oracle 4719776 1 0 13:24:34 - 0:00 /oracle/grid/app/product/11.2.0/gi/bin/oraagent.bin[/code]
切换到oracle用户，手工执行oraagent：
su - oracle
/oracle/grid/app/product/11.2.0/gi/bin/oraagent

报错很多lib文件没权限
2个节点
su - root
chmod 755 /oracle/grid/app/product/11.2.0/gi/lib/lib*

2个节点
su - oracle
/oracle/grid/app/product/11.2.0/gi/bin/oraagent
/oracle/grid/app/product/11.2.0/gi/bin/oraagent

之后可以正常启动数据库：
[code]grid@hxcrmdb1:/oracle/grid$srvctl start database -d hxcrm
grid@hxcrmdb1:/oracle/grid$crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE hxcrmdb1
ONLINE ONLINE hxcrmdb2
ora.asm
OFFLINE OFFLINE hxcrmdb1
OFFLINE OFFLINE hxcrmdb2
ora.gsd
OFFLINE OFFLINE hxcrmdb1
OFFLINE OFFLINE hxcrmdb2
ora.net1.network
ONLINE ONLINE hxcrmdb1
ONLINE ONLINE hxcrmdb2
ora.ons
ONLINE ONLINE hxcrmdb1
ONLINE ONLINE hxcrmdb2
ora.registry.acfs
OFFLINE OFFLINE hxcrmdb1
OFFLINE OFFLINE hxcrmdb2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE hxcrmdb2
ora.cvu
1 ONLINE ONLINE hxcrmdb2
ora.hxcrm.db
1 ONLINE ONLINE hxcrmdb1 Open
2 ONLINE ONLINE hxcrmdb2 Open
ora.hxcrmdb1.vip
1 ONLINE ONLINE hxcrmdb1
ora.hxcrmdb2.vip
1 ONLINE ONLINE hxcrmdb2
ora.oc4j
1 ONLINE ONLINE hxcrmdb2
ora.scan1.vip
1 ONLINE ONLINE hxcrmdb2[/code]