(1)一次失败的割接经历：麒麟linux6.5+oracle 12c rac打PSU补丁

1.项目要求：

根据安全扫描漏洞整改要求，需对某系统（麒麟linux6.5+oracle12c rac）打220118版本的PSU补丁。

系统状态、磁盘空间等等检查都OK，根据保守经验，申请停机时间3小时
由于在9i/10g/11g/12c/19c rac+centos/redhat/solaris/aix等环境都有过打补丁经验。应该说这次打补丁差不多十拿九稳的事情，没想到很快就被打脸了:)
曾经在19c rac打过补丁踩坑记录，此时没有参考意义:(https://www.modb.pro/doc/46086

2.实施准备：

2.1介质主备：

下载PSU最新补丁集：Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文档 ID 2118136.2)
下载最新opatch工具：How To Download And Install The Latest OPatch(6880880) Version (Doc ID 274526.1)

2.2两节点备份权限：

根据：Script to capture and restore file permission in a directory (for eg. ORACLE_HOME) (Doc ID 1515018.1)

chmod 755 permission.pl 
./permission.pl /u01/app/12.2.0/grid
cp restore-perm-二-9月-20-14-22-39-2022.cmd r.cmd

2.3两节点备份文件（注意保证空间足够）：

cd /u01/app
tar -cvf grid.tar ./12.2.0 ./grid --exclude=./grid/admin --exclude=./grid/diag --exclude=./12.2.0/grid/network/log --exclude=./12.2.0/grid/log --exclude=./12.2.0/grid/rdbms/audit
tar -cvf oracle.tar ./oracle --exclude=./oracle/admin --exclude=./oracle/diag
tar -cvf oraInventory.tar ./oraInventory

2.4opatch工具解压和替换：

先/etc/profile增加环境变量，方便root用户执行和查看命令：

vi /etc/profile
export PATH=/u01/app/12.2.0/grid/OPatch:/u01/app/12.2.0/grid/bin:$PATH;
source /etc/profile

再替换oracle和grid下opatch工具：

cd /home/oracle/soft/2022
unzip p6880880_122010_Linux-x86-64-OPatch 12.2.0.1.32.zip
rm -rf /u01/app/oracle/product/12.2.0/db_1/OPatch
cp -rp OPatch /u01/app/oracle/product/12.2.0/db_1/
chown -R oracle:oinstall /u01/app/oracle/product/12.2.0/db_1/OPatch
rm -rf /u01/app/12.2.0/grid/OPatch
cp -rf OPatch /u01/app/12.2.0/grid/
chown -R grid:oinstall /u01/app/12.2.0/grid/OPatch
export PATH=$PATH:/u01/app/12.2.0/grid/OPatch
opatch version

2.5补丁冲突检查

根据readme执行即可，如下示例,省略部分：

grid用户下核查：
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33678030
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33116894
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/26839277
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33610989

oracle用户下核查：
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128

root用户下预执行分析：（此次预执行分析都成功，但实际执行却报错，继续往下看）
opatchauto apply /home/oracle/soft/2022/33559966/33583921 -analyze

3.正式实施：

root用户下在节点1先执行：

opatchauto apply /home/oracle/soft/2022/33559966/33583921

可以看到db补丁成功，但开始应用grid补丁的时候失败了：


根据错误位置信息查看内容：

执行chmod -R 755 /u01/app/12.2.0/grid/jdk，然后opatchauto resume
继续报错jdk目录权限：
查到相关问题参考：


赋权chown -R grid:oinstall /u01/app/12.2.0/grid/jdk，然后opatchauto resume,还是报错bin、lib等目录权限，约9千行的错误：

根据错误日志，提取grep Destionation err.txt > 1.txt，然后排序去重，过滤出300多行的文件。
编辑脚本执行赋权：

chown grid:oinstall /u01/app/12.2.0/grid/bin/HiveMetadata.jar
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsccm
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsdriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfshanfs
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsload
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsregistrymount
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_initializer
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_monitor
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_preapply
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_transport
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsreplcrs
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrm
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/acfssinglefsmount
chown grid:oinstall /u01/app/12.2.0/grid/bin/advmccb
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdboot
chown grid:oinstall /u01/app/12.2.0/grid/bin/afddriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdload
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool
chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/appagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg
chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/asmproxy
chown grid:oinstall /u01/app/12.2.0/grid/bin/bdschecksw
chown grid:oinstall /u01/app/12.2.0/grid/bin/cdpc.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutlo.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutls.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clscfg.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsdiaglsnr.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsecho.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clsid.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/clssproxy.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_getperm.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_profile.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_register.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_relocate.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_setperm.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_start.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stat.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stop.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_unregister.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crscdpd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsctl.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsdiag.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/crsrename.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/crstmpl.scr
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdagent
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdmonitor
chown grid:oinstall /u01/app/12.2.0/grid/bin/cssvfupgd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/dbfs_client
chown grid:oinstall /u01/app/12.2.0/grid/bin/diagcollection.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/diagsnap_setup.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/diskmon
chown grid:oinstall /u01/app/12.2.0/grid/bin/dsml2ldif
chown grid:oinstall /u01/app/12.2.0/grid/bin/emcrsp.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evminfo.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmlogger.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmkbin.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmklib.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmpost.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmshow.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmsort.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evmwatch.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/evt.sh
chown grid:oinstall /u01/app/12.2.0/grid/bin/fmputl
chown grid:oinstall /u01/app/12.2.0/grid/bin/ghappctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/gipcd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd
chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnpd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnptool.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/gsd
chown grid:oinstall /u01/app/12.2.0/grid/bin/jwcctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/kfod
chown grid:oinstall /u01/app/12.2.0/grid/bin/kgpmon
chown grid:oinstall /u01/app/12.2.0/grid/bin/lcsscan
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapadd
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapaddmt
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapbind
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapcompare
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapdelete
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmoddn
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodify
chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodifymt
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsdb.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes
chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxchknlb
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxegen
chown grid:oinstall /u01/app/12.2.0/grid/bin/lxinst
chown grid:oinstall /u01/app/12.2.0/grid/bin/mdnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/mkstore
chown grid:oinstall /u01/app/12.2.0/grid/bin/mtactl
chown grid:oinstall /u01/app/12.2.0/grid/bin/ochadctl.pl
chown grid:oinstall /u01/app/12.2.0/grid/bin/oclskd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oclumon.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrcheck.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrconfig.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrdump.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssdrim.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/octssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/odig
chown grid:oinstall /u01/app/12.2.0/grid/bin/odig.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/odnsd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/ohasd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oifcfg.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd
chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/okadriverstate
chown grid:oinstall /u01/app/12.2.0/grid/bin/okaload
chown grid:oinstall /u01/app/12.2.0/grid/bin/okaroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/olfscmd
chown grid:oinstall /u01/app/12.2.0/grid/bin/olfsroot
chown grid:oinstall /u01/app/12.2.0/grid/bin/ologgerd
chown grid:oinstall /u01/app/12.2.0/grid/bin/olsnodes.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oprocd
chown grid:oinstall /u01/app/12.2.0/grid/bin/oraagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnfs
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd
chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/oranetmonitor.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/orapki
chown grid:oinstall /u01/app/12.2.0/grid/bin/orarootagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/osysmond.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/owm
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgeut
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgevtf
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgmain
chown grid:oinstall /u01/app/12.2.0/grid/bin/racgvip
chown grid:oinstall /u01/app/12.2.0/grid/bin/schema
chown grid:oinstall /u01/app/12.2.0/grid/bin/sclsspawn
chown grid:oinstall /u01/app/12.2.0/grid/bin/scriptagent.bin
chown grid:oinstall /u01/app/12.2.0/grid/bin/srvconfig
chown grid:oinstall /u01/app/12.2.0/grid/bin/usrvip
chown grid:oinstall /u01/app/12.2.0/grid/bin/xml
chown grid:oinstall /u01/app/12.2.0/grid/crs/install/s_crsconfig_defs
chown grid:oinstall /u01/app/12.2.0/grid/crs/install/tfa_setup
chown grid:oinstall /u01/app/12.2.0/grid/jlib/gns.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/netcfg.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvm.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmasm.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmhas.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster-common.jar
chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster.jar
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsdriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfshanfs.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsregistrymount.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsreplcrs.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfssinglefsmount.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/acfstoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/afddriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/afdtoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdafd.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdambr.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdamdu.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdbase.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdpasswd.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdqg.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdshare.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdsys.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdug.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcommand.xml
chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_common.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_lib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/libCurveFit.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libNeticaJ.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagent12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagfw12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libagtsh.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclnt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclntsh12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmperl12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_asym.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_non_fips.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcha12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclient12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so.12.1
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntshcore.so.12.1
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsce12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclscred12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsnsjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsr12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsra12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcommon12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcore12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcorejava.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf_mdb12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.sig
chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libctx12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libctxc12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libdbcfg12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libeons.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libeonsserver.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgeneric12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgns12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libgnsjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libhasgen12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libjagent12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libldapclnt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libn12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnbeq12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libncrypt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnhost12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnid.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnzst12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libntcps12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocr12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrb12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrutl12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/liboevm.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/liborabz2.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libordsdo12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libowm2.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12_pic.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libpsa12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/librdjni12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libserver12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpg.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpr.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libskjcx12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsql12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsqlplus.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmcred12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmhas12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmocr12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libuini12.so
chown grid:oinstall /u01/app/12.2.0/grid/lib/libxml12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libzt12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/libztkg12.a
chown grid:oinstall /u01/app/12.2.0/grid/lib/naect.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/naeet.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/nautab.o
chown grid:oinstall /u01/app/12.2.0/grid/lib/okadriverstate.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/okaload.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okaroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/okatoolsdriver.sh
chown grid:oinstall /u01/app/12.2.0/grid/lib/olfscmd.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/olfsroot.pl
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsdriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfshanfs.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsregistrymount.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfssinglefsmount.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afddriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okadriverstate.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaload.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaroot.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_acfslib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_afdlib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_okalib.pm
chown grid:oinstall /u01/app/12.2.0/grid/lib/s_jwcctl_lib.pm
chown grid:oinstall /u01/app/12.2.0/grid/srvm/jlib/srvctl.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable.pm
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable/Wrap.pm
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/collectfiles.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/discover_ora_stack.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/getppid.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/metric_iorm.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/patchtfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/profiling.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfasetup.pl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin/tnt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf/tnt.prop.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-cli-1.2.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-lang3-3.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-logging-1.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/tnt.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xmlparserv2.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xternal.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/input
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/init.tfa.tmpl
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.aix
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.hpux
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.linux
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.sunos
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.conf
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/roottfa.sh
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/.buildid
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/usableports.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/RATFA.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/Symlink.so
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/commons-io-2.1.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jdev-rt.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-4.1.27.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-5.0.84.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jewt4.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jsch-0.1.51.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/ojdbc5.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/share.jar
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/public.jks
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/date_patterns.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignore_extensions.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignorefiles.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/mask_strings.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/problemset.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/searchStrings.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/tasks.xml
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.jks
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.md5
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_directories.txt
chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_setup.txt

mkdir -p /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin
mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/jdk
mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/javafx
mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited
mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited
mkdir -p /u01/app/12.2.0/grid/jdk/legal/jdk
mkdir -p /u01/app/12.2.0/grid/jdk/legal/javafx

chown -R grid:oinstall /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/jdk
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/javafx
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/jdk
chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/javafx
chown -R grid:oinstall /u01/app/12.2.0/grid/crs/install
chown grid:oinstall /u01/app/12.2.0/grid/bin
chown grid:oinstall /u01/app/12.2.0/grid/lib

继续opatchauto resume，还是报错

进程占用处理：

fuser -ck /u01/app/12.2.0/grid/bin/oracle            
fuser -ck /u01/app/12.2.0/grid/lib/libclntsh.so.12.1 
fuser -ck /u01/app/12.2.0/grid/lib/libasmclntsh12.so 

多次执行fuser删进程，确认没有进程了即可
继续opatchauto resume，提示grid patch成功，但crsctl start crs最终还是报错了：


第一个节点到此失败后尝试各种办法，均无法启动crsd集群服务，
比如根据CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)尝试：

/u01/app/12.2.0/grid/crs/install/rootcrs.sh -prepatch
/u01/app/12.2.0/grid/crs/install/rootcrs.sh -postpatch
opatchauto resume还是失败

由于时间紧迫，没有多余的时间再做研究分析，只能宣告此次打补丁失败，决定回退。

ps -ef|grep d.bin后kill掉相关进程。
然后tar -xvf解压原先备份的tar包。重启集群服务恢复正常。通知应用检查恢复业务。

收集好错误日志信息，第二天开始复盘分析问题。
由于手头没有麒麟linux6.5介质，所以尝试先用centos6.10搭相同的12c rac环境，补丁版本保持一致。（高效的做法：虚机环境+克隆+nfs共享存储+系统快照）
相关参考：
克隆的网卡问题https://sqlora.blog.csdn.net/article/details/126869607
nfs搭asmhttps://sqlora.blog.csdn.net/article/details/126951455
没想到同样的步骤很顺利的打上了PSU补丁，没有遇到任何错误。
此时怀疑麒麟系统是否存在兼容性bug问题？由于用户要求做安全整改必须打上补丁，所以只能找用户要来了麒麟linux6.5系统介质。
同样搭好相同的12c rac环境，补丁版本保持一致。希望能重现故障。

功夫不负有心人，节点1操作果然重现了生产环境打补丁的故障现象。于是小心翼翼的分析每一步错误，并做好记录。查阅大量资料及测试，最终顺利打上了集群补丁。

测试验证过程中，同样遇到了上述问题crs无法启动，通过查阅oracle资料发现：
CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)其中说到节点两边补丁级别不一致的情况下，肯定是无法启动的，所以决定放弃尝试，先不管。

继续给第二个节点补丁

opatchauto apply /home/oracle/soft/2022/33559966/33583921

报错如下：

然后检查两节点的补丁情况，级别和补丁完全一致，但状态不对。
应该是【NORMAL】，但都显示【ROLLING PATCH】
虽然状态不对，但此时发现节点1和节点2的集群服务均是正常的，检查命令：

crsctl query crs activeversion -f
kfod op=patches
kfod op=patchlvl
crsctl check crs
crsctl stat res -t -init
crsctl stat res -t 

[grid@rac1 ~]$ kfod op=patches
---------------
List of Patches
===============
26839277
33116894
33587128
33610989
33678030
[grid@rac1 ~]$ kfod op=patchlvl
-------------------
Current Patch level
===================
3975995681
[grid@rac1 ~]$ ssh rac2
Last login: Tue Sep 20 05:30:51 2022 from rac1
[grid@rac2 ~]$ kfod op=patches
---------------
List of Patches
===============
26839277
33116894
33587128
33610989
33678030
[grid@rac2 ~]$ kfod op=patchlvl
-------------------
Current Patch level
===================
3975995681

既然版本和补丁都一致，剩下就是状态的处理了：
参考：Software Patch Mismatch After Applying Bundle Patch or PSU on Grid Home and the Cluster Upgrade State is [ROLLING PATCH] (Doc ID 2176790.1)
在节点1执行如下命令后，节点1、2状态均恢复正常：

处理操作命令如下：

[root@rac1 ~]# clscfg -patch
clscfg: -patch mode specified
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
[root@rac1 ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [927320293].
[root@rac1 ~]# crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [3975995681].
[root@rac1 ~]# su - grid
[grid@rac2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681].
[grid@rac2 ~]$ exit
logout
Connection to rac2 closed.
[grid@rac1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681].

终于在麒麟虚机环境上第一次圆满打上了补丁。
经过反复改进尝试，最终整理了更顺畅的打补丁步骤【麒麟Linux6.5+Oracle 12c rac psu 190716->220118验证测试】