12C Extended RAC高可用环境详细搭建步骤

这是一套真实在运行的7X24交易系统，截止目前已经稳定运行2年多，数据库SLA 100%。
由于篇幅较长，部分数据库基础配置信息略，大家可以主要关注一下多路径及ASM相关配置。

一、环境规划

二、操作系统环境配置
1、安装必须的RPM包
（略）

2、修改内核参数文件（/etc/sysctl.conf,部分略）
因为使用haip，当有多张网卡时，必须配置如下参数，否则只能启动一个节点：
net.ipv4.conf.default.arp_filter = 2
net.ipv4.conf.all.rp_filter = 2

3、配置host(/etc/hosts)
192.168.10.81 RAC1
192.168.10.82 RAC2

192.168.10.83 RAC1-VIP
192.168.10.84 RAC2-VIP

192.168.20.1 RAC1-PRIV1
192.168.20.2 RAC1-PRIV2
192.168.20.3 RAC2-PRIV1
192.168.20.4 RAC2-PRIV2

192.168.10.80 RAC-SCAN

4、配置资源限制文件
（limits.conf,20-nproc.conf,略）

5、修改共享内存大小
（/dev/shm,略）

6、关闭防火墙
（略）

7、配置MTU参数
（MTU=9000，略）

8、创建用户和组
（group(oinstall,asmdba,asmoper,asmadmin,dba),user(grid,oracle),略）

9、创建相关目录
（grid,grid_base,oracle_home,oracle_base,略）

10、配置时间同步
（禁用NTP，采用oracle的CTSS，略）

11、配置oracle，grid用户环境变量
（略）

12、配置互信
（略）

13、关闭selinux
(/etc/selinux.conf,略)

14、配置共享磁盘
安装多路径软件：
yum -y install device-mapper*

修改多路径配置文件：
defaults {
user_friendly_names yes
}

blacklist {
devnode “^sda”
}

devices {
device {
vendor “COMPELNT”
product “Compellent Vol”
path_grouping_policy multibus
getuid_callout “/lib/udev/scsi_id --whitelisted --device=/dev/%n”
path_selector “round-robin 0”
hardware_handler “0”
failback immediate
path_checker tur
rr_weight uniform
rr_min_io_rq 1000
no_path_retry fail
}
}

multipaths {
multipath {
alias DATA_A
wwid “36000d310057c1a000000000000000003”
uid 0
gid 0
mode 0600
}
multipath {
alias FRA_A
wwid “36000d310057c1a000000000000000004”
uid 0
gid 0
mode 0600
}
multipath {
alias REDO_A
wwid “36000d310057c1a000000000000000006”
uid 0
gid 0
mode 0600
}
multipath {
alias OCR_A
wwid “36000d310057c1a000000000000000005”
uid 0
gid 0
mode 0600
}
multipath {
alias DATA_B
wwid “36000d310057caa000000000000000003”
uid 0
gid 0
mode 0600
}
multipath {
alias FRA_B
wwid “36000d310057caa000000000000000004”
uid 0
gid 0
mode 0600
}
multipath {
alias REDO_B
wwid “36000d310057caa000000000000000006”
uid 0
gid 0
mode 0600
}
multipath {
alias OCR_B
wwid “36000d310057caa000000000000000005”
uid 0
gid 0
mode 0600
}
multipath {
alias T_OCR
wwid “36000d310057c1a000000000000000008”
uid 0
gid 0
mode 0600
}
multipath {
alias GIMR_A
wwid “36000d310057c1a000000000000000007”
uid 0
gid 0
mode 0600
}
multipath {
alias GIMR_B
wwid “36000d310057caa000000000000000007”
uid 0
gid 0
mode 0600
}
}

启动多路径服务并配置自动启动：
service multipathd start
hkconfig multipathd on

UDEV权限绑定并配置开机自动启动：
[root@hkora1 ~]# vi /etc/udev/rules.d/99-asm.rules
KERNEL==“dm-[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
KERNEL==“dm-1[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
KERNEL==“dm-2[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
KERNEL==“dm-3[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
KERNEL==“dm-4[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
KERNEL==“dm-5[0-9]”, OWNER=“grid”, GROUP=“asmadmin”, MODE=“0660”
[root@hkora1 ~]# start_udev
Starting udev: [ OK ]
配置udev开机启动：vi /etc/rc.local
/sbin/start_udev

查看多路径：
DATA_B (36000d310057caa000000000000000003) dm-9 COMPELNT,Compellent Vol
size=2.0T features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:3:1 sdf 8:80 active ready running - 16:0:2:1 sdr 65:16 active ready running
REDO_B (36000d310057caa000000000000000006) dm-11 COMPELNT,Compellent Vol
size=200G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:3:5 sdh 8:112 active ready running - 16:0:2:5 sdt 65:48 active ready running
DATA_A (36000d310057c1a000000000000000003) dm-0 COMPELNT,Compellent Vol
size=2.0T features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:0:1 sdb 8:16 active ready running - 16:0:0:1 sdn 8:208 active ready running
REDO_A (36000d310057c1a000000000000000006) dm-2 COMPELNT,Compellent Vol
size=200G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:0:5 sdd 8:48 active ready running - 16:0:0:5 sdp 8:240 active ready running
GIMR_B (36000d310057caa000000000000000007) dm-10 COMPELNT,Compellent Vol
size=100G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:3:3 sdg 8:96 active ready running - 16:0:2:3 sds 65:32 active ready running
GIMR_A (36000d310057c1a000000000000000007) dm-1 COMPELNT,Compellent Vol
size=100G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:0:3 sdc 8:32 active ready running - 16:0:0:3 sdo 8:224 active ready running
T_OCR (36000d310057c1a000000000000000008) dm-6 COMPELNT,Compellent Vol
size=10G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 16:0:1:6 sdy 65:128 active ready running - 15:0:1:6 sdm 8:192 active ready running
OCR_B (36000d310057caa000000000000000005) dm-8 COMPELNT,Compellent Vol
size=10G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:2:4 sdj 8:144 active ready running - 16:0:3:4 sdv 65:80 active ready running
OCR_A (36000d310057c1a000000000000000005) dm-5 COMPELNT,Compellent Vol
size=10G features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:1:4 sdl 8:176 active ready running - 16:0:1:4 sdx 65:112 active ready running
FRA_B (36000d310057caa000000000000000004) dm-7 COMPELNT,Compellent Vol
size=2.0T features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:2:2 sdi 8:128 active ready running - 16:0:3:2 sdu 65:64 active ready running
FRA_A (36000d310057c1a000000000000000004) dm-4 COMPELNT,Compellent Vol
size=2.0T features=‘0’ hwhandler=‘0’ wp=rw
-+- policy='round-robin 0' prio=1 status=active |- 15:0:1:2 sdk 8:160 active ready running - 16:0:1:2 sdw 65:96 active ready running

15、安装asmlib，创建asm磁盘
安装asmlib：
yum -y install kmod-oracleasm-2.0.8-15.el6_9.x86_64.rpm oracleasmlib-2.0.4-1.el6.x86_64.rpm oracleasm-support-2.1.8-1.el6.x86_64.rpm

配置asmlib：
[root@hkrac1 opt]# /etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets (’[]’). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
scanning the system for Oracle ASMLib disks: [ OK ]

创建asm disks：
[root@hkrac1 ~]# oracleasm createdisk DATA_A /dev/mapper/DATA_A
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk DATA_B /dev/mapper/DATA_B
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk FRA_A /dev/mapper/FRA_A
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk FRA_B /dev/mapper/FRA_B
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk REDO_A /dev/mapper/REDO_A
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk REDO_B /dev/mapper/REDO_B
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk OCR_A /dev/mapper/OCR_A
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk OCR_B /dev/mapper/OCR_B
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk GIMR_A /dev/mapper/GIMR_A
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk GIMR_B /dev/mapper/GIMR_B
Writing disk header: done
Instantiating disk: done
[root@hkrac1 ~]# oracleasm createdisk T_OCR /dev/mapper/T_OCR
Writing disk header: done
Instantiating disk: done

节点2同步：
[root@hkrac2 ~]# oracleasm scandisks

查看创建好的asm disks：
[root@hkrac2 ~]# oracleasm listdisks
DATA_A
DATA_B
FRA_A
FRA_B
OCR_A
OCR_B
REDO_A
REDO_B
T_OCR
GIMR_A
GIMR_B

配置oracleasm配置文件：
vi /etc/sysconfig/oracleasm
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=“mpath dm”
ORACLEASM_SCANEXCLUDE=“sd”
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

重启asmlib服务：
service oracleasm restart

查看配置是否生效：
oracleasm configure

16、挂在NFS
安装NFS客户端包：
yum install -y nfs-utils

查看远程NFS共享的盘：
showmount -e 192.168.10.51

挂载共享盘：
mount -t nfs -o nolock,rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 192.168.30.51:/data_nfs2 /ocr

配置开机自动挂载：
vi /etc/fstab
192.168.30.51:/data_nfs2 /ocr nfs rw,bg,hard,intr,rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600 0 0

用dd命令写一个空文件：
dd if=/dev/zero of=/ocr/vote_disk bs=1M count=51200

17、开启linux大页
计算需要的页数：
linux一个大页的大小为2M，开启大页的总内存应该比sga_max_size稍稍大一点，比如
sga_max_size=180g，则：hugepages > (180*1024)/2 = 92160

配置sysctl.conf文件，添加：
vm.nr_hugepages = 92200

配置/etc/security/limit.conf，添加（比sga_max_size稍大，以K为单位）：
oracle soft memlock 188800000
oracle hard memlock 188800000

禁用透明大页（注意添加位置）:
添加 transparent_hugepage=never
vi /etc/grub.conf
Once modified the line will read similar to the following example:
title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/
transparent_hugepage=never
initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img

重启操作系统，验证大页是否被使用以及透明大页是否关闭：
cat /proc/meminfo | grep -i Huge

AnonHugePages为0，则表示已经关闭透明大页
HugePages_Total:1030
HugePages_Free: 774
HugePages_Rsvd: 769
表示已经开启大页

三、安装GI软件

四、配置asm磁盘组

五、数据库软件安装
（略）

六、创建数据库
（略）

七、迁移votedisk和ocr等
1、迁移votdisk
crsctl replace votedisk +OCR
crsctl query css votedisk

2、迁移ocr
ocrconfig -add +OCR
ocrconfig -delete +TOCR
ocrcheck

3、grid用户迁移spfile
asmcmd spget
asmcmd spmove ‘+TOCR/hkrac-cluster/ASMPARAMETERFILE/registry.253.826898607’ ‘+OCR/hkrac-cluster/asmparameterfile/spfileasm.ora’

4、grid用户迁移PWDFILE
asmcmd pwmove --asm -f +TOCR/orapwasm +OCR/orapwasm
srvctl config asm

至此，整套集群搭建完毕，两台服务器，两台交换机，两台存储任意节点故障，集群均能正常对外提供服务。