最近在巡检中，发现一个rac库的2节点crs的alert日志里报下面的错：

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-26 18:55:21.370

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb.db with CARDINALITY_ID=2' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 02:56:47.904

[/u01/app/11.2.0/grid/bin/orarootagent.bin(8585442)]CRS-5818:Aborted command 'check for resource: ora.crsd 1 1' for resource 'ora.crsd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/sldb2/agent/ohasd/orarootagent_root/orarootagent_root.log.

2019-05-27 02:56:48.294

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 02:58:21.459

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 02:58:21.462

[crsd(11927682)]CRS-1015:A request to terminate the Cluster Ready Service on node sldb1 completed successfully. Details at (:PROCR00000:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 02:58:28.278

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 02:58:28.386

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb.db with CARDINALITY_ID=2' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 10:59:56.515

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 11:01:29.627

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 11:01:29.631

[crsd(11927682)]CRS-1015:A request to terminate the Cluster Ready Service on node sldb1 completed successfully. Details at (:PROCR00000:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 11:01:36.601

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 11:01:36.925

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb.db with CARDINALITY_ID=2' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 19:02:51.686

[/u01/app/11.2.0/grid/bin/orarootagent.bin(8585442)]CRS-5818:Aborted command 'check for resource: ora.crsd 1 1' for resource 'ora.crsd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/sldb2/agent/ohasd/orarootagent_root/orarootagent_root.log.

2019-05-27 19:03:03.809

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 19:04:36.925

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 19:04:36.926

[crsd(11927682)]CRS-1015:A request to terminate the Cluster Ready Service on node sldb1 completed successfully. Details at (:PROCR00000:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 19:04:42.569

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-27 19:04:42.598

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb.db with CARDINALITY_ID=2' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 03:06:11.440

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 03:07:44.583

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 03:07:44.584

[crsd(11927682)]CRS-1015:A request to terminate the Cluster Ready Service on node sldb1 completed successfully. Details at (:PROCR00000:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 03:07:51.793

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 03:07:51.853

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb.db with CARDINALITY_ID=2' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 11:08:55.440

[/u01/app/11.2.0/grid/bin/orarootagent.bin(8585442)]CRS-5818:Aborted command 'check for resource: ora.crsd 1 1' for resource 'ora.crsd'. Details at (:CRSAGF00113:) in /u01/app/11.2.0/grid/log/sldb2/agent/ohasd/orarootagent_root/orarootagent_root.log.

2019-05-28 11:09:19.092

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 11:10:52.216

[crsd(11927682)]CRS-1008:Node sldb1 is not responding to OCR requests. Details in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 11:10:52.220

[crsd(11927682)]CRS-1015:A request to terminate the Cluster Ready Service on node sldb1 completed successfully. Details at (:PROCR00000:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 11:10:57.706

[crsd(11927682)]CRS-2750:Cardinality violation detected on server 'sldb1', resource 'ora.sldb1.vip' is in unexpected state. Details at (:CRSPE00104:) in /u01/app/11.2.0/grid/log/sldb2/crsd/crsd.log.

2019-05-28 11:10:57.971

一节点alert里有大量下面语句生成：

crs-2406:the cluster time synchronization service timed out on host sldb1

crs-2406:the cluster time synchronization service timed out on host sldb1

crs-2406:the cluster time synchronization service timed out on host sldb1