asm_diskstring设置错误cssd无法启动恢复

数据库技术加油站 2023-08-29

183

1、背景

以前的同事上周五在某人民医院原地升级11.2.0.4 rac到19.20 rac，升级完成后当天晚上以失败告终，于是回退到了11.2.0.4版本，周六、周日系统通过11.2.0.4运行平稳，直到今天早上8点左右全院系统由卡慢转变到宕机7个小时...

同事邮件事后回复内容：早上快9点的时候医院反馈全院锁表卡顿，无法登录系统，远程查看存在大量的gc buffer busy release等待事件，系统整体hang住，无法通过杀掉进程的方式恢复业务，建议重启数据库，重启数据库因为大量的等待，也非常缓慢，半小时内无法完成数据库关闭，于是分别重启服务器，重启服务器后rac系统无法启动。

2、处理经过

8点多-10点10左右: 前同事做了如上操作，杀进程、重启数据库、重启服务器...

10点20-11点00左右: 我远程过去发现cssd进程无法启动，通过批量kill会话kill -9杀掉/u01，开了几个会话看日志，由于对这个环境不了解、用户又比较紧迫，看了下/u01下面环境目录有11g和19c的，做了第一个操作：清理crs和重配crs：

./rootcrs.pl -deconfig -force -verbose

./rootcrs.pl -deconfig -force -verbose -lastnode

/u01/11.2.4/grid/root.sh --尝试重新配置

[root@hisdb1 ohasd]# u01/11.2.4/grid/root.sh 
Performing root user operation for Oracle 11g 


The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  u01/11.2.4/grid


Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Entries will be added to the etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: u01/11.2.4/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to oracle-ohasd.service
CRS-2672: Attempting to start 'ora.mdnsd' on 'hisdb1'
CRS-2676: Start of 'ora.mdnsd' on 'hisdb1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hisdb1'
CRS-2676: Start of 'ora.gpnpd' on 'hisdb1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb1'
CRS-2672: Attempting to start 'ora.gipcd' on 'hisdb1'
CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hisdb1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hisdb1'
CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb1'
CRS-2676: Start of 'ora.diskmon' on 'hisdb1' succeeded
CRS-2676: Start of 'ora.cssd' on 'hisdb1' succeeded
Start of resource "ora.cssd" failed              --发现该进程还是无法启动
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb1'
CRS-2672: Attempting to start 'ora.gipcd' on 'hisdb1'
CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hisdb1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hisdb1'
CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb1'
CRS-2676: Start of 'ora.diskmon' on 'hisdb1' succeeded
CRS-2674: Start of 'ora.cssd' on 'hisdb1' failed
CRS-2679: Attempting to clean 'ora.cssd' on 'hisdb1'
CRS-2681: Clean of 'ora.cssd' on 'hisdb1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'hisdb1'
CRS-2677: Stop of 'ora.gipcd' on 'hisdb1' succeeded
CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start Cluster Synchorinisation Service in clustered mode at u01/11.2.4/grid/crs/install/crsconfig_lib.pm line 1294.
/u01/11.2.4/grid/perl/bin/perl -I/u01/11.2.4/grid/perl/lib -I/u01/11.2.4/grid/crs/install u01/11.2.4/grid/crs/install/rootcrs.pl execution failed

通过crsctl stat res -t -init查看还是卡在ora.cssd进程上，于是手工执行打开，crsctl start res ora.cssd -init，依然报错；

11点20-12点00左右: 用户急切需要进行恢复，停了几个小时机了，我通过手工的方式mount磁盘组也不行，报ora-15040 ora-15017,看元数据等也是正常的；然后和用户沟通恢复方案：1、抽出asm数据异机或本地恢复 2、重建crs；

12点00-15点10左右: 通过商量，在一节点的机器上进行数据恢复，将rac变为单机，抽取数据到本地文件系统，然后执行nomount\mount\open动作，然后oracle用户下netca创建一个监听，修改/etc/hosts文件指向scan，同时修改ip恢复业务；

* 节点2 关闭crs，在节点1操作，将rac改为单机


[root@hisdb1 rmanbak]# crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@hisdb1 rmanbak]# 
[root@hisdb1 rmanbak]# crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@hisdb1 rmanbak]# 
[root@hisdb1 rmanbak]# su - oracle
Last login: Mon Aug 28 11:41:01 CST 2023 on pts/1
[oracle@hisdb1 ~]$ cd $ORACLE_HOME/rdbms/lib
[oracle@hisdb1 lib]$ pwd
/u01/app/oracle/product/11.2.4/db_1/rdbms/lib
[oracle@hisdb1 lib]$ make -f ins_rdbms.mk rac_off
rm -f u01/app/oracle/product/11.2.4/db_1/lib/libskgxp11.so
cp u01/app/oracle/product/11.2.4/db_1/lib//libskgxpg.so u01/app/oracle/product/11.2.4/db_1/lib/libskgxp11.so
rm -f u01/app/oracle/product/11.2.4/db_1/lib/libskgxn2.so
cp u01/app/oracle/product/11.2.4/db_1/lib//libskgxns.so \
      u01/app/oracle/product/11.2.4/db_1/lib/libskgxn2.so
/usr/bin/ar d u01/app/oracle/product/11.2.4/db_1/rdbms/lib/libknlopt.a kcsm.o
/usr/bin/ar cr u01/app/oracle/product/11.2.4/db_1/rdbms/lib/libknlopt.a u01/app/oracle/product/11.2.4/db_1/rdbms/lib/ksnkcs.o 
[oracle@hisdb1 lib]$ make -f ins_rdbms.mk ioracle
chmod 755 u01/app/oracle/product/11.2.4/db_1/bin


 - Linking Oracle 
rm -f u01/app/oracle/product/11.2.4/db_1/rdbms/lib/oracle
gcc  -o u01/app/oracle/product/11.2.4/db_1/rdbms/lib/oracle -m64 -z noexecstack -L/u01/app/oracle/product/11.2.4/db_1/rdbms/lib/ -L/u01/app/oracle/product/11.2.4/db_1/lib/ -L/u01/app/oracle/product/11.2.4/db_1/lib/stubs/   -Wl,-E u01/app/oracle/product/11.2.4/db_1/rdbms/lib/opimai.o u01/app/oracle/product/11.2.4/db_1/rdbms/lib/ssoraed.o u01/app/oracle/product/11.2.4/db_1/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive u01/app/oracle/product/11.2.4/db_1/lib/nautab.o u01/app/oracle/product/11.2.4/db_1/lib/naeet.o u01/app/oracle/product/11.2.4/db_1/lib/naect.o u01/app/oracle/product/11.2.4/db_1/lib/naedhs.o u01/app/oracle/product/11.2.4/db_1/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if usr/bin/ar tv u01/app/oracle/product/11.2.4/db_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f u01/app/oracle/product/11.2.4/db_1/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f u01/app/oracle/product/11.2.4/db_1/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat u01/app/oracle/product/11.2.4/db_1/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat u01/app/oracle/product/11.2.4/db_1/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat u01/app/oracle/product/11.2.4/db_1/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat u01/app/oracle/product/11.2.4/db_1/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if usr/bin/ar tv u01/app/oracle/product/11.2.4/db_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/oracle/product/11.2.4/db_1/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat u01/app/oracle/product/11.2.4/db_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/11.2.4/db_1/lib -lm    `cat u01/app/oracle/product/11.2.4/db_1/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/11.2.4/db_1/lib
test ! -f u01/app/oracle/product/11.2.4/db_1/bin/oracle || (\
   mv -f u01/app/oracle/product/11.2.4/db_1/bin/oracle u01/app/oracle/product/11.2.4/db_1/bin/oracleO &&\
   chmod 600 u01/app/oracle/product/11.2.4/db_1/bin/oracleO )
mv u01/app/oracle/product/11.2.4/db_1/rdbms/lib/oracle u01/app/oracle/product/11.2.4/db_1/bin/oracle
chmod 6751 u01/app/oracle/product/11.2.4/db_1/bin/oracle

* 抽取脚本之前发过，公众号里面有，将asm数据抽取出来之后遇到一个情况，system和sysaux不分表空间数据文件有多个，在执行alter database rename file '' to '';在思考该怎么写；
* 另外由于文件很多，asm实例又无法打开，无法查询别名信息，我通过扫描别名目录，将别名取出来进行对比，然后编写修改omf的路径到文件系统：
name:HISDB;fnum:4294967295;finc:4294967295
name:zl9indexcis02.dbf;fnum:268;finc:1099325663
name:zlmlog01.dbf;fnum:270;finc:1099325663
name:zlmlog.dbf;fnum:271;finc:1099325663
name:zl9indexcis01.dbf;fnum:272;finc:1099325701
name:zl9indexhis.dbf;fnum:273;finc:1099325701
name:zl9expense.dbf;fnum:274;finc:1099325703
name:zl9indexcis.dbf;fnum:275;finc:1099325703
name:zl9eprdat.dbf;fnum:277;finc:1099325739
name:zl9indexhis01.dbf;fnum:278;finc:1099325741
name:zl9cisrec.dbf;fnum:279;finc:1099325741
name:zl9patient.dbf;fnum:280;finc:1099325777
name:zl9expense01.dbf;fnum:281;finc:1099325777
name:zl9eprlob.dbf;fnum:282;finc:1099325779
name:zl9cisrec01.dbf;fnum:285;finc:1099325809
name:zl9medlst.dbf;fnum:286;finc:1099325813
name:zl9baseitem.dbf;fnum:287;finc:1099325813
name:zl9eprdat01.dbf;fnum:288;finc:1099325825
name:zl9indexcis03.dbf;fnum:290;finc:1099325827
name:zltoolstbs.dbf;fnum:291;finc:1099325827
name:zl9indexmdr.dbf;fnum:292;finc:1099325827
name:zl9devrec.dbf;fnum:293;finc:1099325827
name:zl9devuse.dbf;fnum:295;finc:1099325829
name:zl9medday.dbf;fnum:296;finc:1099325831
name:zl9cisaudit.dbf;fnum:297;finc:1099325831
name:zl9indexhis02.dbf;fnum:298;finc:1099325833
name:zlpacsbiztab.dbf;fnum:299;finc:1099325835
name:zl9indexdev.dbf;fnum:300;finc:1099325835
name:zl9mtlrec.dbf;fnum:301;finc:1099325835
name:zlpacsbizindex.dbf;fnum:302;finc:1099325835
name:zlmsgdata.dbf;fnum:303;finc:1099325837
name:zl9indexmtl.dbf;fnum:304;finc:1099325839
name:zl9medrec.dbf;fnum:305;finc:1099325845
name:zlbak2011.dbf;fnum:306;finc:1099325849
name:zl9medbase.dbf;fnum:307;finc:1099325849
name:zl9mtlbase.dbf;fnum:308;finc:1099325853
name:zl9devbase.dbf;fnum:309;finc:1099325859
name:zl9duerec.dbf;fnum:310;finc:1099325863
name:zlpacsbizxml.dbf;fnum:311;finc:1099325871
name:zlpacsbasetab.dbf;fnum:312;finc:1099325873
name:zlpacsbaseindex.dbf;fnum:313;finc:1099325875
name:zltoolstmp.dbf;fnum:319;finc:1099387241
name:undotbs2;fnum:324;finc:1099389191
name:DATAFILE;fnum:4294967295;finc:4294967295
name:CONTROLFILE;fnum:4294967295;finc:4294967295
name:ONLINELOG;fnum:4294967295;finc:4294967295
name:TEMPFILE;fnum:4294967295;finc:4294967295
name:PARAMETERFILE;fnum:4294967295;finc:4294967295
name:spfilehisdb.ora;fnum:286;finc:1102346709
name:CHANGETRACKING;fnum:4294967295;finc:4294967295
name:zlmlog.dbf;fnum:265;finc:1102328489
name:zlmlog01.dbf;fnum:262;finc:1102328489
name:zl9indexcis01.dbf;fnum:261;finc:1102328527
name:zl9indexcis.dbf;fnum:284;finc:1102328527
name:zl9indexhis.dbf;fnum:259;finc:1102328527
name:zl9expense.dbf;fnum:264;finc:1102328527
name:zl9indexhis01.dbf;fnum:258;finc:1102328567
name:zl9cisrec.dbf;fnum:276;finc:1102328567
name:zl9eprdat.dbf;fnum:256;finc:1102328567
name:zl9patient.dbf;fnum:269;finc:1102328603
name:zl9eprlob.dbf;fnum:289;finc:1102328605
name:zl9expense01.dbf;fnum:294;finc:1102328605
name:zl9cisrec01.dbf;fnum:291;finc:1102328635
name:zl9medlst.dbf;fnum:311;finc:1102328641
name:zl9baseitem.dbf;fnum:299;finc:1102328645
name:zl9eprdat01.dbf;fnum:302;finc:1102328653
name:zlmlog02.dbf;fnum:312;finc:1102328653
name:zl9indexcis03.dbf;fnum:313;finc:1102328655
name:zl9indexmdr.dbf;fnum:303;finc:1102328657
name:zl9devrec.dbf;fnum:306;finc:1102328661
name:zl9indexdev.dbf;fnum:280;finc:1102328661
name:zl9mtlrec.dbf;fnum:301;finc:1102328661
name:zl9medbase.dbf;fnum:308;finc:1102328661
name:zl9indexhis02.dbf;fnum:305;finc:1102328663
name:zlpacsbaseindex.dbf;fnum:286;finc:1102328665
name:zl9duerec.dbf;fnum:296;finc:1102328665
name:zl9medrec.dbf;fnum:307;finc:1102328667
name:zlpacsbiztab.dbf;fnum:304;finc:1102328667
name:zltoolstbs.dbf;fnum:292;finc:1102328667
name:zl9medday.dbf;fnum:298;finc:1102328669
name:zlpacsbizindex.dbf;fnum:278;finc:1102328669
name:zlmsgdata.dbf;fnum:273;finc:1102328669
name:zlbak2011.dbf;fnum:300;finc:1102328673
name:zl9devbase.dbf;fnum:290;finc:1102328679
name:zl9cisaudit.dbf;fnum:268;finc:1102328683
name:zlpacsbasetab.dbf;fnum:272;finc:1102328685
name:zl9devuse.dbf;fnum:275;finc:1102328755
name:zlpacsbizxml.dbf;fnum:281;finc:1102328757
name:zl9indexmtl.dbf;fnum:274;finc:1102328759
name:zl9mtlbase.dbf;fnum:282;finc:1102328785
name:TEMP;fnum:320;finc:1102325695
name:TEMP;fnum:281;finc:1102346593
name:TEMP;fnum:342;finc:1102374881
name:ZLTOOLSTMP;fnum:324;finc:1102340563
name:ZLTOOLSTMP01.dbf;fnum:324;finc:1102340563
name:TEMP;fnum:327;finc:1102340569
name:TEMP01.dbf;fnum:327;finc:1102340569
name:ZLTOOLSTMP;fnum:343;finc:1102374881
name:ZLMLOG;fnum:271;finc:1099325663
name:ZL9INDEXCIS;fnum:272;finc:1099325701
name:ZL9INDEXHIS;fnum:273;finc:1099325701
name:ZL9EXPENSE;fnum:274;finc:1099325703
name:ZL9INDEXCIS;fnum:275;finc:1099325703
name:SYSTEM;fnum:276;finc:1099325739
name:ZL9EPRDAT;fnum:277;finc:1099325739
name:ZL9INDEXHIS;fnum:278;finc:1099325741
name:ZL9CISREC;fnum:279;finc:1099325741
name:ZL9PATIENT;fnum:280;finc:1099325777
name:ZL9EXPENSE;fnum:281;finc:1099325777
name:ZL9EPRLOB;fnum:282;finc:1099325779
name:UNDOTBS1;fnum:283;finc:1099325779
name:USERS;fnum:284;finc:1099325801
name:ZL9CISREC;fnum:285;finc:1099325809
name:ZL9MEDLST;fnum:286;finc:1099325813
name:ZL9BASEITEM;fnum:287;finc:1099325813
name:ZL9EPRDAT;fnum:288;finc:1099325825
name:RMAN;fnum:289;finc:1099325825
name:ZL9INDEXCIS;fnum:290;finc:1099325827
name:ZLTOOLSTBS;fnum:291;finc:1099325827
name:ZL9INDEXMDR;fnum:292;finc:1099325827
name:ZL9DEVREC;fnum:293;finc:1099325827
name:EXAMPLE;fnum:294;finc:1099325829
name:ZL9DEVUSE;fnum:295;finc:1099325829
name:ZL9MEDDAY;fnum:296;finc:1099325831
name:ZL9CISAUDIT;fnum:297;finc:1099325831
name:ZL9INDEXHIS;fnum:298;finc:1099325833
name:ZLPACSBIZTAB;fnum:299;finc:1099325835
name:ZL9INDEXDEV;fnum:300;finc:1099325835
name:ZL9MTLREC;fnum:301;finc:1099325835
name:ZLPACSBIZINDEX;fnum:302;finc:1099325835
name:ZLMSGDATA;fnum:303;finc:1099325837
name:ZL9INDEXMTL;fnum:304;finc:1099325839
name:ZL9MEDREC;fnum:305;finc:1099325845
name:ZLBAK2011;fnum:306;finc:1099325849
name:ZL9MEDBASE;fnum:307;finc:1099325849
name:ZL9MTLBASE;fnum:308;finc:1099325853
name:ZL9DEVBASE;fnum:309;finc:1099325859
name:ZL9DUEREC;fnum:310;finc:1099325863
name:ZLPACSBIZXML;fnum:311;finc:1099325871
name:ZLPACSBASETAB;fnum:312;finc:1099325873
name:ZLPACSBASEINDEX;fnum:313;finc:1099325875
name:UNDOTBS2;fnum:324;finc:1099389191
name:Current;fnum:260;finc:1099307517
name:Current;fnum:323;finc:1102325689
name:Current;fnum:266;finc:1102346587
name:group_1;fnum:322;finc:1102325691
name:group_2;fnum:321;finc:1102325691
name:group_3;fnum:316;finc:1102325823
name:group_4;fnum:315;finc:1102325823
name:group_11;fnum:288;finc:1102338871
name:group_12;fnum:277;finc:1102338871
name:group_13;fnum:310;finc:1102338873
name:group_14;fnum:295;finc:1102338873
name:group_21;fnum:279;finc:1102375079
name:redo21.log;fnum:279;finc:1102375079
name:group_22;fnum:285;finc:1102375081
name:redo22.log;fnum:285;finc:1102375081
name:group_23;fnum:309;finc:1102375081
name:redo23.log;fnum:309;finc:1102375081
name:group_24;fnum:293;finc:1102375083
name:redo24.log;fnum:293;finc:1102375083
name:group_1;fnum:263;finc:1102346589
name:group_2;fnum:292;finc:1102346591
name:group_3;fnum:278;finc:1102346709
name:group_4;fnum:272;finc:1102346709
name:group_11;fnum:338;finc:1102374875
name:group_12;fnum:339;finc:1102374875
name:group_13;fnum:340;finc:1102374877
name:group_14;fnum:341;finc:1102374877
name:group_1;fnum:351;finc:1142801175
name:group_2;fnum:352;finc:1142801175
name:group_3;fnum:353;finc:1142801175
name:group_4;fnum:354;finc:1142801177
name:group_5;fnum:355;finc:1142801185
name:group_6;fnum:356;finc:1142801185
name:group_7;fnum:357;finc:1142801185
name:group_8;fnum:358;finc:1142801185
name:SYSTEM;fnum:270;finc:1102346455
name:SYSAUX;fnum:271;finc:1102346455
name:UNDOTBS1;fnum:326;finc:1102325541
name:USERS;fnum:325;finc:1102325541
name:UNDOTBS2;fnum:317;finc:1102325781
name:UNDOTBS1;fnum:318;finc:1102346455
name:USERS;fnum:283;finc:1102346455
name:UNDOTBS2;fnum:304;finc:1102346671
name:ZL9INDEXCIS;fnum:273;finc:1102347467
name:zl9indexcis02.dbf;fnum:273;finc:1102347467
name:SYSAUX;fnum:312;finc:1102347467
name:ZLMLOG;fnum:262;finc:1102347467
name:zlmlog.dbf;fnum:262;finc:1102347467
name:ZLMLOG;fnum:265;finc:1102347467
name:zlmlog01.dbf;fnum:265;finc:1102347467
name:ZL9INDEXCIS;fnum:300;finc:1102347503
name:zl9indexcis01.dbf;fnum:300;finc:1102347503
name:ZL9INDEXHIS;fnum:269;finc:1102347505
name:UNDOTBS1;fnum:257;finc:1102328605
name:zl9indexhis.dbf;fnum:269;finc:1102347505
name:ZL9EXPENSE;fnum:301;finc:1102347505
name:USERS;fnum:319;finc:1102328629
name:zl9expense.dbf;fnum:301;finc:1102347505
name:ZL9INDEXCIS;fnum:282;finc:1102347505
name:zl9indexcis.dbf;fnum:282;finc:1102347505
name:SYSTEM;fnum:307;finc:1102347541
name:ZL9EPRDAT;fnum:311;finc:1102347543
name:zl9eprdat.dbf;fnum:311;finc:1102347543
name:ZL9INDEXHIS;fnum:298;finc:1102347543
name:zl9indexhis01.dbf;fnum:298;finc:1102347543
name:ZL9CISREC;fnum:308;finc:1102347543
name:zl9cisrec.dbf;fnum:308;finc:1102347543
name:ZL9PATIENT;fnum:274;finc:1102347577
name:zl9patient.dbf;fnum:274;finc:1102347577
name:ZL9EXPENSE;fnum:303;finc:1102347579
name:zl9expense01.dbf;fnum:303;finc:1102347579
name:ZL9EPRLOB;fnum:305;finc:1102347581
name:zl9eprlob.dbf;fnum:305;finc:1102347581
name:UNDOTBS1;fnum:258;finc:1102347583
name:USERS;fnum:259;finc:1102347621
name:ZL9CISREC;fnum:280;finc:1102347631
name:zl9cisrec01.dbf;fnum:280;finc:1102347631
name:ZL9MEDLST;fnum:313;finc:1102347631
name:zl9medlst.dbf;fnum:313;finc:1102347631
name:ZL9BASEITEM;fnum:314;finc:1102347637
name:zl9baseitem.dbf;fnum:314;finc:1102347637
name:ZL9INDEXCIS;fnum:261;finc:1102347645
name:zl9indexcis03.dbf;fnum:261;finc:1102347645
name:ZL9EPRDAT;fnum:284;finc:1102347649
name:zl9eprdat01.dbf;fnum:284;finc:1102347649
name:ZL9INDEXMDR;fnum:294;finc:1102347649
name:zl9indexmdr.dbf;fnum:294;finc:1102347649
name:UNDOTBS2;fnum:297;finc:1102375181
name:spfile;fnum:286;finc:1102346709
name:undotbs2;fnum:297;finc:1102375181
name:ZLMLOG;fnum:264;finc:1102347649
name:zlmlog02.dbf;fnum:264;finc:1102347649
name:RMAN;fnum:289;finc:1102347651
name:ZL9INDEXDEV;fnum:302;finc:1102347651
name:zl9indexdev.dbf;fnum:302;finc:1102347651
name:ZL9MEDBASE;fnum:256;finc:1102347651
name:zl9medbase.dbf;fnum:256;finc:1102347651
name:EXAMPLE;fnum:296;finc:1102347651
name:ZL9DEVREC;fnum:275;finc:1102347655
name:zl9devrec.dbf;fnum:275;finc:1102347655
name:ZL9MTLREC;fnum:306;finc:1102347655
name:zl9mtlrec.dbf;fnum:306;finc:1102347655
name:ZL9INDEXHIS;fnum:290;finc:1102347655
name:zl9indexhis02.dbf;fnum:290;finc:1102347655
name:ZL9DUEREC;fnum:291;finc:1102347657
name:zl9duerec.dbf;fnum:291;finc:1102347657
name:ZLPACSBASEINDEX;fnum:276;finc:1102347657
name:zlpacsbaseindex.dbf;fnum:276;finc:1102347657
name:ZL9MEDDAY;fnum:268;finc:1102347659
name:zl9medday.dbf;fnum:268;finc:1102347659
name:ZLTOOLSTBS;fnum:299;finc:1102347659
name:zltoolstbs.dbf;fnum:299;finc:1102347659
name:ZL9MEDREC;fnum:267;finc:1102347659
name:zl9medrec.dbf;fnum:267;finc:1102347659
name:ZLPACSBIZTAB;fnum:287;finc:1102347661
name:zlpacsbiztab.dbf;fnum:287;finc:1102347661
name:ZLPACSBIZINDEX;fnum:328;finc:1102347661
name:zlpacsbizindex.dbf;fnum:328;finc:1102347661
name:ZLMSGDATA;fnum:329;finc:1102347663
name:zlmsgdata.dbf;fnum:329;finc:1102347663
name:ZLBAK2011;fnum:330;finc:1102347665
name:zlbak2011.dbf;fnum:330;finc:1102347665
name:ZL9DEVBASE;fnum:331;finc:1102347673
name:zl9devbase.dbf;fnum:331;finc:1102347673
name:ZLPACSBASETAB;fnum:332;finc:1102347677
name:zlpacsbasetab.dbf;fnum:332;finc:1102347677
name:ZL9CISAUDIT;fnum:333;finc:1102347677
name:zl9cisaudit.dbf;fnum:333;finc:1102347677
name:ZL9DEVUSE;fnum:334;finc:1102347747
name:zl9devuse.dbf;fnum:334;finc:1102347747
name:ZL9INDEXMTL;fnum:335;finc:1102347751
name:zl9indexmtl.dbf;fnum:335;finc:1102347751
name:ZLPACSBIZXML;fnum:336;finc:1102347753
name:zlpacsbizxml.dbf;fnum:336;finc:1102347753
name:ZL9MTLBASE;fnum:337;finc:1102347775
name:zl9mtlbase.dbf;fnum:337;finc:1102347775
name:zltoolstmp.dbf;fnum:343;finc:1102374881
name:ZL9EXPENSE;fnum:344;finc:1102693901
name:zl9expense02.dbf;fnum:344;finc:1102693901
name:VINTERFACEDATA;fnum:345;finc:1112373199
name:VINTERFACEDATA.DBF;fnum:345;finc:1112373199
name:ZLMLOG;fnum:346;finc:1126003287
name:zlmlog03.dbf;fnum:346;finc:1126003287
name:USERS;fnum:347;finc:1130840501
name:users01.dbf;fnum:347;finc:1130840501
name:ZL9INDEXCIS;fnum:348;finc:1133857975
name:zl9indexcis04.dbf;fnum:348;finc:1133857975
name:ZL9INDEXHIS;fnum:349;finc:1133858195
name:zl9indexhis03.dbf;fnum:349;finc:1133858195
name:ZLMLOG;fnum:350;finc:1138525227
name:zlmlog04.dbf;fnum:350;finc:1138525227
name:blktrace.dbf;fnum:359;finc:1145797597
name:ctf;fnum:359;finc:1145797597


























+DATA/hisdb/datafile/system.307.1102347541 》》alter database rename file '+DATA/hisdb/datafile/system.307.1102347541' to '/rmanbak/hisdbdata/SYSTEM.307';
+DATA/hisdb/datafile/sysaux.312.1102347467     alter database rename file '+DATA/hisdb/datafile/sysaux.312.1102347467' to '/rmanbak/hisdbdata/SYSAUX.312';
+DATA/hisdb/datafile/undotbs1.258.1102347583   alter database rename file '+DATA/hisdb/datafile/undotbs1.258.1102347583' to '/rmanbak/hisdbdata/UNDOTBS1.258';
+DATA/hisdb/datafile/users.259.1102347621          alter database rename file '+DATA/hisdb/datafile/users.259.1102347621' to '/rmanbak/hisdbdata/USERS.259';
+DATA/hisdb/datafile/zlmlog02.dbf                          alter database rename file '+DATA/hisdb/datafile/zlmlog02.dbf' to '/rmanbak/hisdbdata/zlmlog02.dbf.264';
+DATA/hisdb/datafile/zltoolstbs.dbf                         alter database rename file '+DATA/hisdb/datafile/zltoolstbs.dbf' to '/rmanbak/hisdbdata/zltoolstbs.dbf.299';
+DATA/hisdb/datafile/zl9baseitem.dbf                    alter database rename file '+DATA/hisdb/datafile/zl9baseitem.dbf' to '/rmanbak/hisdbdata/zl9baseitem.dbf.314';
+DATA/hisdb/datafile/zl9patient.dbf                       alter database rename file '+DATA/hisdb/datafile/zl9patient.dbf' to '/rmanbak/hisdbdata/zl9patient.dbf.274';
+DATA/hisdb/datafile/zl9expense.dbf                      alter database rename file '+DATA/hisdb/datafile/zl9expense.dbf' to '/rmanbak/hisdbdata/zl9expense.dbf.301';
+DATA/hisdb/datafile/zl9medlst.dbf                         alter database rename file '+DATA/hisdb/datafile/zl9medlst.dbf' to '/rmanbak/hisdbdata/zl9medlst.dbf.313';
+DATA/hisdb/datafile/zl9duerec.dbf                        alter database rename file '+DATA/hisdb/datafile/zl9duerec.dbf' to '/rmanbak/hisdbdata/zl9duerec.dbf.291';


NAME
--------------------------------------------------------------------------------
+DATA/hisdb/datafile/zl9cisrec.dbf                          alter database rename file '+DATA/hisdb/datafile/zl9cisrec.dbf' to '/rmanbak/hisdbdata/zl9cisrec.dbf.308';
+DATA/hisdb/datafile/zl9eprlob.dbf                        alter database rename file '+DATA/hisdb/datafile/zl9eprlob.dbf' to '/rmanbak/hisdbdata/zl9eprlob.dbf.305';
+DATA/hisdb/datafile/zl9eprdat.dbf                        alter database rename file '+DATA/hisdb/datafile/zl9eprdat.dbf' to '/rmanbak/hisdbdata/zl9eprdat.dbf.311';
+DATA/hisdb/datafile/zl9cisaudit.dbf                     alter database rename file '+DATA/hisdb/datafile/zl9cisaudit.dbf' to '/rmanbak/hisdbdata/zl9cisaudit.dbf.333';
+DATA/hisdb/datafile/zl9indexhis.dbf                    alter database rename file '+DATA/hisdb/datafile/zl9indexhis.dbf' to '/rmanbak/hisdbdata/zl9indexhis.dbf.269';
+DATA/hisdb/datafile/zl9indexcis.dbf                     alter database rename file '+DATA/hisdb/datafile/zl9indexcis.dbf' to '/rmanbak/hisdbdata/zl9indexcis.dbf.282';  --
+DATA/hisdb/datafile/zlbak2011.dbf                      alter database rename file '+DATA/hisdb/datafile/zlbak2011.dbf' to '/rmanbak/hisdbdata/zlbak2011.dbf.330';
+DATA/hisdb/datafile/zl9devbase.dbf                     alter database rename file '+DATA/hisdb/datafile/zl9devbase.dbf' to '/rmanbak/hisdbdata/zl9devbase.dbf.331';
+DATA/hisdb/datafile/zl9devrec.dbf                       alter database rename file '+DATA/hisdb/datafile/zl9devrec.dbf' to '/rmanbak/hisdbdata/zl9devrec.dbf.275';
+DATA/hisdb/datafile/zl9devuse.dbf                      alter database rename file '+DATA/hisdb/datafile/zl9devuse.dbf' to '/rmanbak/hisdbdata/zl9devuse.dbf.334';
+DATA/hisdb/datafile/zl9indexdev.dbf                  alter database rename file '+DATA/hisdb/datafile/zl9indexdev.dbf' to '/rmanbak/hisdbdata/zl9indexdev.dbf.302';


NAME
--------------------------------------------------------------------------------
+DATA/hisdb/datafile/zl9mtlbase.dbf                     alter database rename file '+DATA/hisdb/datafile/zl9mtlbase.dbf' to '/rmanbak/hisdbdata/zl9mtlbase.dbf.337';
+DATA/hisdb/datafile/zl9mtlrec.dbf                        alter database rename file '+DATA/hisdb/datafile/zl9mtlrec.dbf' to '/rmanbak/hisdbdata/zl9mtlrec.dbf.306';
+DATA/hisdb/datafile/zl9indexmtl.dbf                    alter database rename file '+DATA/hisdb/datafile/zl9indexmtl.dbf' to '/rmanbak/hisdbdata/zl9indexmtl.dbf.335';
+DATA/hisdb/datafile/zl9medbase.dbf                   alter database rename file '+DATA/hisdb/datafile/zl9medbase.dbf' to '/rmanbak/hisdbdata/zl9medbase.dbf.256';
+DATA/hisdb/datafile/zl9medrec.dbf                      alter database rename file '+DATA/hisdb/datafile/zl9medrec.dbf' to '/rmanbak/hisdbdata/zl9medrec.dbf.267';
+DATA/hisdb/datafile/zl9medday.dbf                     alter database rename file '+DATA/hisdb/datafile/zl9medday.dbf' to '/rmanbak/hisdbdata/zl9medday.dbf.268';
+DATA/hisdb/datafile/zl9indexmdr.dbf                  alter database rename file '+DATA/hisdb/datafile/zl9indexmdr.dbf' to '/rmanbak/hisdbdata/zl9indexmdr.dbf.294';
+DATA/hisdb/datafile/zl9indexcis01.dbf                alter database rename file '+DATA/hisdb/datafile/zl9indexcis01.dbf' to '/rmanbak/hisdbdata/zl9indexcis01.dbf.300';
+DATA/hisdb/datafile/zl9indexhis01.dbf                alter database rename file '+DATA/hisdb/datafile/zl9indexhis01.dbf' to '/rmanbak/hisdbdata/zl9indexhis01.dbf.298';
+DATA/hisdb/datafile/example.296.1102347651  alter database rename file '+DATA/hisdb/datafile/example.296.1102347651' to '/rmanbak/hisdbdata/EXAMPLE.296';
+DATA/hisdb/datafile/rman.289.1102347651        alter database rename file '+DATA/hisdb/datafile/rman.289.1102347651' to '/rmanbak/hisdbdata/RMAN.289';


NAME
--------------------------------------------------------------------------------
+DATA/hisdb/datafile/zlpacsbasetab.dbf               alter database rename file '+DATA/hisdb/datafile/zlpacsbasetab.dbf' to '/rmanbak/hisdbdata/zlpacsbasetab.dbf.332';
+DATA/hisdb/datafile/zlpacsbaseindex.dbf            alter database rename file '+DATA/hisdb/datafile/zlpacsbaseindex.dbf' to '/rmanbak/hisdbdata/zlpacsbaseindex.dbf.276';
+DATA/hisdb/datafile/zlpacsbiztab.dbf                  alter database rename file '+DATA/hisdb/datafile/zlpacsbiztab.dbf' to '/rmanbak/hisdbdata/zlpacsbiztab.dbf.287';
+DATA/hisdb/datafile/zlpacsbizindex.dbf             alter database rename file '+DATA/hisdb/datafile/zlpacsbizindex.dbf' to '/rmanbak/hisdbdata/zlpacsbizindex.dbf.328';
+DATA/hisdb/datafile/zlpacsbizxml.dbf                 alter database rename file '+DATA/hisdb/datafile/zlpacsbizxml.dbf' to '/rmanbak/hisdbdata/zlpacsbizxml.dbf.336';
+DATA/hisdb/datafile/zlmsgdata.dbf                    alter database rename file '+DATA/hisdb/datafile/zlmsgdata.dbf' to '/rmanbak/hisdbdata/zlmsgdata.dbf.329';
+DATA/hisdb/datafile/zl9indexcis02.dbf               alter database rename file '+DATA/hisdb/datafile/zl9indexcis02.dbf' to '/rmanbak/hisdbdata/zl9indexcis02.dbf.273';
+DATA/hisdb/datafile/zl9cisrec01.dbf                    alter database rename file '+DATA/hisdb/datafile/zl9cisrec01.dbf' to '/rmanbak/hisdbdata/zl9cisrec01.dbf.280';
+DATA/hisdb/datafile/zl9expense01.dbf               alter database rename file '+DATA/hisdb/datafile/zl9expense01.dbf ' to '/rmanbak/hisdbdata/zl9expense01.dbf.303'; 《》
+DATA/hisdb/datafile/zl9eprdat01.dbf                alter database rename file '+DATA/hisdb/datafile/zl9eprdat01.dbf' to '/rmanbak/hisdbdata/zl9eprdat01.dbf.284';
+DATA/hisdb/datafile/zlmlog.dbf                        alter database rename file '+DATA/hisdb/datafile/zlmlog.dbf ' to '/rmanbak/hisdbdata/zlmlog.dbf.262'; 《》


NAME
--------------------------------------------------------------------------------
+DATA/hisdb/datafile/zlmlog01.dbf                    alter database rename file '+DATA/hisdb/datafile/zlmlog01.dbf' to '/rmanbak/hisdbdata/zlmlog01.dbf.265';
+DATA/hisdb/datafile/zl9indexhis02.dbf             alter database rename file '+DATA/hisdb/datafile/zl9indexhis02.dbf' to '/rmanbak/hisdbdata/zl9indexhis02.dbf.290';
+DATA/hisdb/datafile/zl9indexcis03.dbf             alter database rename file '+DATA/hisdb/datafile/zl9indexcis03.dbf' to '/rmanbak/hisdbdata/zl9indexcis03.dbf.261';
+DATA/hisdb/datafile/undotbs2                         alter database rename file '+DATA/hisdb/datafile/undotbs2' to '/rmanbak/hisdbdata/undotbs2.297';
+DATA/hisdb/datafile/zl9expense02.dbf              alter database rename file '+DATA/hisdb/datafile/zl9expense02.dbf' to '/rmanbak/hisdbdata/zl9expense02.dbf.344';
+DATA/hisdb/datafile/vinterfacedata.dbf             alter database rename file '+DATA/hisdb/datafile/vinterfacedata.dbf' to '/rmanbak/hisdbdata/VINTERFACEDATA.DBF.345';
+DATA/hisdb/datafile/zlmlog03.dbf                    alter database rename file '+DATA/hisdb/datafile/zlmlog03.dbf' to '/rmanbak/hisdbdata/zlmlog03.dbf.346';
+DATA/hisdb/datafile/users01.dbf                        alter database rename file '+DATA/hisdb/datafile/users01.dbf' to '/rmanbak/hisdbdata/users01.dbf.347';
+DATA/hisdb/datafile/zl9indexcis04.dbf              alter database rename file '+DATA/hisdb/datafile/zl9indexcis04.dbf' to '/rmanbak/hisdbdata/zl9indexcis04.dbf.348';
+DATA/hisdb/datafile/zl9indexhis03.dbf               alter database rename file '+DATA/hisdb/datafile/zl9indexhis03.dbf' to '/rmanbak/hisdbdata/zl9indexhis03.dbf.349';
+DATA/hisdb/datafile/zlmlog04.dbf                   alter database rename file '+DATA/hisdb/datafile/zlmlog04.dbf' to '/rmanbak/hisdbdata/zlmlog04.dbf.350';


col file# for 999
col name for a60
select file#,name from v$datafile order by 2;
-----------------------------------------------------------------------------
+DATA/hisdb/datafile/zl9expense01.dbf     alter database rename file '+DATA/hisdb/datafile/zl9expense01.dbf' to '/rmanbak/hisdbdata/zl9expense01.dbf.303'; 《》
+DATA/hisdb/datafile/zlmlog.dbf                alter database rename file '+DATA/hisdb/datafile/zlmlog.dbf' to '/rmanbak/hisdbdata/zlmlog.dbf.262'; 《》


select checkpoint_change#,file# from v$datafile_header; --看文件头检查点是否一致；如果一致说明是一批文件，如果跨度大，检查文件是否reneme错误；也可以通过我之前的bbed批量去扫这些文件头，获取resetlogs和上一次resetlogs以及scn信息；
SQL> select group#,thread# from v$Log;


    GROUP#    THREAD#
---------- ----------
  11      1
  12      1
  13      1
  14      1
  21      2
  22      2
  23      2
  24      2


col group# for 9999
col member for a60
select group#,member from v$logfile;


GROUP# MEMBER
------ ------------------------------------------------------------
    14 +DATA/hisdb/onlinelog/group_14.341.1102374877  alter database rename file '+DATA/hisdb/onlinelog/group_14.341.1102374877' to '/rmanbak/hisdbdata/log/group_14.341';
    14 +ARCH/hisdb/onlinelog/group_14.440.1102374877  alter database rename file '+ARCH/hisdb/onlinelog/group_14.440.1102374877' to '/rmanbak/hisdbdata/log/group_14.440';
    13 +DATA/hisdb/onlinelog/group_13.340.1102374877  alter database rename file '+DATA/hisdb/onlinelog/group_13.340.1102374877' to '/rmanbak/hisdbdata/log/group_13.340';
    13 +ARCH/hisdb/onlinelog/group_13.446.1102374877  alter database rename file '+ARCH/hisdb/onlinelog/group_13.446.1102374877' to '/rmanbak/hisdbdata/log/group_13.446';
    12 +DATA/hisdb/onlinelog/group_12.339.1102374875  alter database rename file '+DATA/hisdb/onlinelog/group_12.339.1102374875' to '/rmanbak/hisdbdata/log/group_12.339';
    12 +ARCH/hisdb/onlinelog/group_12.453.1102374875  alter database rename file '+ARCH/hisdb/onlinelog/group_12.453.1102374875' to '/rmanbak/hisdbdata/log/group_12.453';
    11 +DATA/hisdb/onlinelog/group_11.338.1102374875  alter database rename file '+DATA/hisdb/onlinelog/group_11.338.1102374875' to '/rmanbak/hisdbdata/log/group_11.338';
    11 +ARCH/hisdb/onlinelog/group_11.437.1102374875  alter database rename file '+ARCH/hisdb/onlinelog/group_11.437.1102374875' to '/rmanbak/hisdbdata/log/group_11.437';
    21 +DATA/hisdb/onlinelog/redo21.log    alter database rename file '+DATA/hisdb/onlinelog/redo21.log' to '/rmanbak/hisdbdata/log/redo21.log.279';
    22 +DATA/hisdb/onlinelog/redo22.log    alter database rename file '+DATA/hisdb/onlinelog/redo22.log' to '/rmanbak/hisdbdata/log/redo22.log.285';
    23 +DATA/hisdb/onlinelog/redo23.log    alter database rename file '+DATA/hisdb/onlinelog/redo23.log' to '/rmanbak/hisdbdata/log/redo23.log.309';


GROUP# MEMBER
------ ------------------------------------------------------------
    24 +DATA/hisdb/onlinelog/redo24.log    alter database rename file '+DATA/hisdb/onlinelog/redo24.log' to '/rmanbak/hisdbdata/log/redo24.log.293';
    21 +ARCH/hisdb/onlinelog/redo21.log    alter database rename file '+ARCH/hisdb/onlinelog/redo21.log' to '/rmanbak/hisdbdata/log/redo21.log.3272';
    22 +ARCH/hisdb/onlinelog/group_22.3315.1102935729  alter database rename file '+ARCH/hisdb/onlinelog/group_22.3315.1102935729' to '/rmanbak/hisdbdata/log/group_22.3315';
    23 +ARCH/hisdb/onlinelog/group_23.3320.1102935821  alter database rename file '+ARCH/hisdb/onlinelog/group_23.3320.1102935821' to '/rmanbak/hisdbdata/log/group_23.3320';
    24 +ARCH/hisdb/onlinelog/group_24.3321.1102935825  alter database rename file '+ARCH/hisdb/onlinelog/group_24.3321.1102935825' to '/rmanbak/hisdbdata/log/group_24.3321';
     1 +DATA/hisdb/onlinelog/group_1.351.1142801175  --这里我没处理standby，因为不需要
     2 +DATA/hisdb/onlinelog/group_2.352.1142801175
     3 +DATA/hisdb/onlinelog/group_3.353.1142801175  
     4 +DATA/hisdb/onlinelog/group_4.354.1142801177
     5 +DATA/hisdb/onlinelog/group_5.355.1142801185
     6 +DATA/hisdb/onlinelog/group_6.356.1142801185


GROUP# MEMBER
------ ------------------------------------------------------------
     7 +DATA/hisdb/onlinelog/group_7.357.1142801185
     8 +DATA/hisdb/onlinelog/group_8.358.1142801185


24 rows selected.


SQL> select name from v$tempfile;


NAME
------------------------------------------------------------
+DATA/hisdb/tempfile/temp.342.1102374881  alter database rename file '+DATA/hisdb/tempfile/temp.342.1102374881' to '/rmanbak/hisdbdata/TEMP.342';
+DATA/hisdb/datafile/zltoolstmp.dbf    alter database rename file '+DATA/hisdb/datafile/zltoolstmp.dbf' to '/rmanbak/hisdbdata/zltoolstmp.dbf.343';
SQL> alter database open;
ORA-38760: This database instance failedtoturnonflashback database


--通过查看闪回区，发现同事在升级时候创建了3个快照，但是现在asm磁盘组由无法读取，闪回日志就无法读取。
--通过drop restore point xxx，失败；
--关闭数据库闪回再开库还是无法解决；
--于是通过重建控制文件方式将数据库进行 open；
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "HISDB" NORESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    MAXINSTANCES 32
    MAXLOGHISTORY 9344
LOGFILE
  GROUP 11 (
    '/rmanbak/hisdbdata/log/group_11.338',
    '/rmanbak/hisdbdata/log/group_11.437'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 12 (
    '/rmanbak/hisdbdata/log/group_12.339',
    '/rmanbak/hisdbdata/log/group_12.453'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 13 (
    '/rmanbak/hisdbdata/log/group_13.340',
    '/rmanbak/hisdbdata/log/group_13.446'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 14 (
    '/rmanbak/hisdbdata/log/group_14.341',
    '/rmanbak/hisdbdata/log/group_14.440'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 21 (
    '/rmanbak/hisdbdata/log/redo21.log.279',
    '/rmanbak/hisdbdata/log/redo21.log.3272'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 22 (
    '/rmanbak/hisdbdata/log/redo22.log.285',
    '/rmanbak/hisdbdata/log/group_22.3315'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 23 (
    '/rmanbak/hisdbdata/log/redo23.log.309',
    '/rmanbak/hisdbdata/log/group_23.3320'
  ) SIZE 200M BLOCKSIZE 512,
  GROUP 24 (
    '/rmanbak/hisdbdata/log/redo24.log.293',
    '/rmanbak/hisdbdata/log/group_24.3321'
  ) SIZE 200M BLOCKSIZE 512
DATAFILE
  '/rmanbak/hisdbdata/SYSTEM.307',
  '/rmanbak/hisdbdata/SYSAUX.312',
  '/rmanbak/hisdbdata/UNDOTBS1.258',
  '/rmanbak/hisdbdata/USERS.259',
  '/rmanbak/hisdbdata/zlmlog02.dbf.264',
  '/rmanbak/hisdbdata/zltoolstbs.dbf.299',
  '/rmanbak/hisdbdata/zl9baseitem.dbf.314',
  '/rmanbak/hisdbdata/zl9patient.dbf.274',
  '/rmanbak/hisdbdata/zl9expense.dbf.301',
  '/rmanbak/hisdbdata/zl9medlst.dbf.313',
  '/rmanbak/hisdbdata/zl9duerec.dbf.291',
  '/rmanbak/hisdbdata/zl9cisrec.dbf.308',
  '/rmanbak/hisdbdata/zl9eprlob.dbf.305',
  '/rmanbak/hisdbdata/zl9eprdat.dbf.311',
  '/rmanbak/hisdbdata/zl9cisaudit.dbf.333',
  '/rmanbak/hisdbdata/zl9indexhis.dbf.269',
  '/rmanbak/hisdbdata/zl9indexcis.dbf.282',
  '/rmanbak/hisdbdata/zlbak2011.dbf.330',
  '/rmanbak/hisdbdata/zl9devbase.dbf.331',
  '/rmanbak/hisdbdata/zl9devrec.dbf.275',
  '/rmanbak/hisdbdata/zl9devuse.dbf.334',
  '/rmanbak/hisdbdata/zl9indexdev.dbf.302',
  '/rmanbak/hisdbdata/zl9mtlbase.dbf.337',
  '/rmanbak/hisdbdata/zl9mtlrec.dbf.306',
  '/rmanbak/hisdbdata/zl9indexmtl.dbf.335',
  '/rmanbak/hisdbdata/zl9medbase.dbf.256',
  '/rmanbak/hisdbdata/zl9medrec.dbf.267',
  '/rmanbak/hisdbdata/zl9medday.dbf.268',
  '/rmanbak/hisdbdata/zl9indexmdr.dbf.294',
  '/rmanbak/hisdbdata/zl9indexcis01.dbf.300',
  '/rmanbak/hisdbdata/zl9indexhis01.dbf.298',
  '/rmanbak/hisdbdata/EXAMPLE.296',
  '/rmanbak/hisdbdata/RMAN.289',
  '/rmanbak/hisdbdata/zlpacsbasetab.dbf.332',
  '/rmanbak/hisdbdata/zlpacsbaseindex.dbf.276',
  '/rmanbak/hisdbdata/zlpacsbiztab.dbf.287',
  '/rmanbak/hisdbdata/zlpacsbizindex.dbf.328',
  '/rmanbak/hisdbdata/zlpacsbizxml.dbf.336',
  '/rmanbak/hisdbdata/zlmsgdata.dbf.329',
  '/rmanbak/hisdbdata/zl9indexcis02.dbf.273',
  '/rmanbak/hisdbdata/zl9cisrec01.dbf.280',
  '/rmanbak/hisdbdata/zl9expense01.dbf.303',
  '/rmanbak/hisdbdata/zl9eprdat01.dbf.284',
  '/rmanbak/hisdbdata/zlmlog.dbf.262',
  '/rmanbak/hisdbdata/zlmlog01.dbf.265',
  '/rmanbak/hisdbdata/zl9indexhis02.dbf.290',
  '/rmanbak/hisdbdata/zl9indexcis03.dbf.261',
  '/rmanbak/hisdbdata/undotbs2.297',
  '/rmanbak/hisdbdata/zl9expense02.dbf.344',
  '/rmanbak/hisdbdata/VINTERFACEDATA.DBF.345',
  '/rmanbak/hisdbdata/zlmlog03.dbf.346',
  '/rmanbak/hisdbdata/users01.dbf.347',
  '/rmanbak/hisdbdata/zl9indexcis04.dbf.348',
  '/rmanbak/hisdbdata/zl9indexhis03.dbf.349',
  '/rmanbak/hisdbdata/zlmlog04.dbf.350'
CHARACTER SET ZHS16GBK
;
RECOVER DATABASE；
ALTER DATABASE OPEN;
ALTER TABLESPACE TEMP ADD TEMPFILE '/rmanbak/hisdbdata/TEMP.342' REUSE autoextend on;
ALTER TABLESPACE ZLTOOLSTMP ADD TEMPFILE '/rmanbak/hisdbdata/zltoolstmp.dbf.343' REUSE autoextend on;


--接着就是netca，修改/etc/hosts和网卡重启恢复了业务；

3、总结

整个过程处理下来还是比较耗费精力，事后回想如果不抽数据怎么恢复呢？因为这个用户的环境确实不太熟悉，加上周一还没进入状态、业务全部停摆等各种压力，心里想得是尽快恢复业务；选择了这种方式；

事后通过回想加测试，我找到了问题，用户使用的是多路径，然后做的裸设备，同事在做升级的时候修改了一个参数asm_diskstring【alter system set asm_diskstring='/dev/dm-*,/dev/raw/raw*'】导致cssd进程无法启动；dm和raw都指向相同磁盘组磁盘，以下是我大致测试过程：

1、gpnp位置：($GRID_HOME/gpnp/<hostname>/profiles/peer/profile.xml).
2、不支持手动修改gpnp profile.xml
3、每当通过“alter system”命令修改 ASM 实例参数“asm_diskstring”时，全局即插即用配置文件都会自动更新。


1）修改asm_diskstring为错误的值
sqlplus / as sysasm
SQL> alter system set asm_diskstring='BAD*' scope=spfile;
当发出以下命令后，集群中所有的profile.xml都会进行更新；
[grid@hisdb01 peer]$ gpnptool get
<orcl:ASM-Profile id="asm" DiscoveryString="BAD*" SPFile="+SYSTEMDG/hisdb-cluster/asmparameterfile/registry.253.1030192111"/>
文件配置文件中的发现字符串.xml将变为：DiscoveryString="BAD*"


2）下次重新启动 CRS 堆栈时
ps -ef | grep d.bin | awk '{print $2}' | xargs kill -9
ps -ef |grep /u01| awk '{print $2}' | xargs kill -9


--ocssd.log 将显示以下错误，并且启动将失败：
[root@hisdb01 cssd]# pwd
/u01/app/11.2.0/grid/log/hisdb01/cssd


...
2023-08-28 22:41:08.581: [    GPNP][1844926208]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_hisdb01" disco "" [f=0 claimed- host: cname: seq: auth:] 
2023-08-28 22:41:08.584: [    GPNP][1844926208]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_hisdb01" disco ""
2023-08-28 22:41:08.584: [    CSSD][1844926208]clssnmReadDiscoveryProfile: voting file discovery string(BAD*)
2023-08-28 22:41:08.584: [    CSSD][1844926208]clssnmvDDiscThread: using discovery string BAD* for initial discovery 
2023-08-28 22:41:08.584: [   SKGFD][1844926208]Discovery with str:BAD*:


2023-08-28 22:41:08.584: [   SKGFD][1844926208]UFS discovery with :BAD*:


2023-08-28 22:41:08.584: [   SKGFD][1844926208]Execute glob on the string /u01/app/11.2.0/grid/dbs/BAD*
2023-08-28 22:41:08.584: [   SKGFD][1844926208]OSS discovery with :BAD*:


2023-08-28 22:41:08.584: [    CSSD][1844926208]clssnmvDiskVerify: Successful discovery of 0 disks
2023-08-28 22:41:08.584: [    CSSD][1844926208]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
...


3）杀掉进程，重启crs独占模式，重建asm参数文件
ps -ef | grep d.bin | awk '{print $2}' | xargs kill -9
ps -ef |grep /u01| awk '{print $2}' | xargs kill -9


【root用户执行以下脚本】
11.2.0.1执行：
$GRID_HOME/bin/crsctl start crs -excl
11.2.0.2以上执行：
$GRID_HOME/bin/crsctl start crs -excl -nocrs


[root@hisdb01 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'hisdb01'
CRS-2676: Start of 'ora.mdnsd' on 'hisdb01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'hisdb01'
CRS-2676: Start of 'ora.gpnpd' on 'hisdb01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'hisdb01'
CRS-2672: Attempting to start 'ora.gipcd' on 'hisdb01'
CRS-2676: Start of 'ora.cssdmonitor' on 'hisdb01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'hisdb01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'hisdb01'
CRS-2672: Attempting to start 'ora.diskmon' on 'hisdb01'
CRS-2676: Start of 'ora.diskmon' on 'hisdb01' succeeded
CRS-2676: Start of 'ora.cssd' on 'hisdb01' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'hisdb01'
CRS-2672: Attempting to start 'ora.ctssd' on 'hisdb01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'hisdb01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'hisdb01'
CRS-2676: Start of 'ora.ctssd' on 'hisdb01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'hisdb01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'hisdb01'
CRS-2676: Start of 'ora.asm' on 'hisdb01' succeeded
[root@hisdb01 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  INTERMEDIATE hisdb01                  OCR not started     
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       hisdb01                                      
ora.crf
      1        OFFLINE OFFLINE                                                   
ora.crsd
      1        OFFLINE OFFLINE                                                   
ora.cssd
      1        ONLINE  ONLINE       hisdb01                                      
ora.cssdmonitor
      1        ONLINE  ONLINE       hisdb01                                      
ora.ctssd
      1        ONLINE  ONLINE       hisdb01                  ACTIVE:0            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        OFFLINE OFFLINE                                                   
ora.gipcd
      1        ONLINE  ONLINE       hisdb01                                      
ora.gpnpd
      1        ONLINE  ONLINE       hisdb01                                      
ora.mdnsd
      1        ONLINE  ONLINE       hisdb01      




从alert_+ASM<x>.log 里面提取初始化参数：--注意asm磁盘的权限，可以移除asm_disksting参数


4）创建一个pfile，然后启动pfile
【grid用户执行以下脚本】
[grid@hisdb01 ~]$ vi /tmp/initasm.txt


*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'


注意：必须正确写上asm_diskstring，不然mount不了磁盘组：
###########
ORA-15032: not all alterations performed
ORA-15017: diskgroup "SYSTEMDG" cannot be mounted
ORA-15040: diskgroup is incomplete




SQL> alter diskgroup systemdg mount;
alter diskgroup systemdg mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "SYSTEMDG" cannot be mounted
ORA-15040: diskgroup is incomplete
###########


[grid@hisdb01 ~]$ sqlplus / as sysasm


SQL*Plus: Release 11.2.0.4.0 Production on Mon Aug 28 23:16:43 2023


Copyright (c) 1982, 2013, Oracle.  All rights reserved.




Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options


SQL> select status from v$instance;


STATUS
------------------------
STARTED


SQL> shutdown abort
ASM instance shutdown


SQL> startup pfile='/tmp/initasm.txt';
ASM instance started


Total System Global Area 1135747072 bytes
Fixed Size        2260728 bytes
Variable Size     1108320520 bytes
ASM Cache       25165824 bytes
ASM diskgroups mounted


SQL> create spfile='+SYSTEMDG' from pfile='/tmp/initasm.txt';


File created.


5）执行完成后，检查profile.xml是否更新($GRID_HOME/gpnp/<hostname>/profiles/peer/profile.xml).
[grid@hisdb01 ~]$ gpnptool get
<orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+SYSTEMDG/hisdb-cluster/asmparameterfile/registry.253.1146094277"/>


--
[grid@hisdb01 ~]$ tail -200f /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
NOTE: updated gpnp profile ASM SPFILE to 
NOTE: updated gpnp profile ASM diskstring: /dev/asm*
NOTE: updated gpnp profile ASM diskstring: /dev/asm*
NOTE: updated gpnp profile ASM SPFILE to +SYSTEMDG/hisdb-cluster/asmparameterfile/registry.253.1146094277


6）替换voting disk到这个磁盘组
[grid@hisdb01 peer]$ crsctl query css votedisk
Located 0 voting disk(s).


[grid@hisdb01 peer]$ crsctl replace votedisk +SYSTEMDG
Successful addition of voting disk fa64bb86fec94ff3bfe9820420289199.
Successful addition of voting disk d800a5435c7b4f18bff39264ef07f422.
Successful addition of voting disk e849a0be17174f5cbf94db10d3021007.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4266: Voting file(s) successfully replaced


[grid@hisdb01 peer]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   fa64bb86fec94ff3bfe9820420289199 (/dev/asmdisk1) [SYSTEMDG]
 2. ONLINE   d800a5435c7b4f18bff39264ef07f422 (/dev/asmdisk2) [SYSTEMDG]
 3. ONLINE   e849a0be17174f5cbf94db10d3021007 (/dev/asmdisk3) [SYSTEMDG]
Located 3 voting disk(s).


7）以root用户身份停止并重新启动 CRS：
# $GRID_HOME/bin/crsctl stop crs -f
# $GRID_HOME/bin/crsctl start crs


8）以root用户身份在其余集群节点上启动 CRS：
# $GRID_HOME/bin/crsctl start crs

人生难免遇上很多选择，每个命令的返回或者每个日志的输出再加上各种环境都可能影响你的判断，一旦选择了一条路走，再尝试另外的路就需要付出更多的时间和命令去弥补；选择没有对与错，只要能迎头直上不退缩，问题终将是会迎刃而解。

文章转载自数据库技术加油站，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

asm_diskstring设置错误cssd无法启动恢复

评论