19C Extended Rac Cluster 高可用架构搭建

MeetDB 2020-05-10

492

Extended Rac是Oracle自身提供的一种同城双活解决方案，解决了机房级故障，单一储存故障的容灾能力。区别于同城ADG主备，它不需要人工介入，提升了容灾水平。

本文旨在模拟搭建19C Extended Rac Cluster 集群环境，让大家有一个基础的了解。

一. 规划

二. 环境初始

三.安装GI

四. 配置asm磁盘组

五. 安装数据库软件

六. 创建数据库

七.校验

一. 规划：

环境简述：使用4个centos 7.6 mini操作系统作为4个数据节点，3个openfiler操作系统提供三个存储。

主机放置规划：

site0: 19C-SStorage0

site1: exn1,exn2,19C-SStorage1

site2: exn3,exn4,19C-SStorage2

IP规划

192.168.1.190   kevin-scan
192.168.1.191   exn1
192.168.1.192   exn2
192.168.1.193   exn3
192.168.1.194   exn4
192.168.1.196   exn1-vip
192.168.1.197   exn2-vip
192.168.1.198   exn3-vip
192.168.1.199   exn4-vip
192.168.2.191   exn1-priv
192.168.2.192   exn2-priv
192.168.2.193   exn3-priv
192.168.2.194   exn4-priv

ASM DISK GROUP规划：

SStorage0（site0 仲裁中心）:

SStorage1(site1):

SStorage2(site2):

网卡Team制作：

示例：exn1做网卡team

Team1:

nmcli device
nmcli con add type team con-name Team1 ifname Team1 config '{"runner": {"name": "activebackup"}}'
nmcli con add type ethernet con-name Team1-slave1 ifname ens32 master Team1
nmcli con add type ethernet con-name Team1-slave2 ifname ens33 master Team1
nmcli connection modify Team1 ipv4.addresses 192.168.1.191/24
nmcli connection modify Team1 ipv4.method manual
nmcli connection up Team1

Team2:

nmcli con add type team con-name Team2 ifname Team2 config '{"runner": {"name": "activebackup"}}'
nmcli con add type ethernet con-name Team2-slave1 ifname ens34 master Team2
nmcli con add type ethernet con-name Team2-slave2 ifname ens35 master Team2
nmcli connection modify Team2 ipv4.addresses 192.168.2.191/24
nmcli connection modify Team2 ipv4.method manual
nmcli connection up Team2

二. 环境初始：（四个节点做相同操作）

1.修改主机hosts文件

[root@exn1 ~]# cat etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.190   kevin-scan
192.168.1.191   exn1
192.168.1.192   exn2
192.168.1.193   exn3
192.168.1.194   exn4
192.168.1.196   exn1-vip
192.168.1.197   exn2-vip
192.168.1.198   exn3-vip
192.168.1.199   exn4-vip
192.168.2.191   exn1-priv
192.168.2.192   exn2-priv
192.168.2.193   exn3-priv
192.168.2.194   exn4-priv

2.关闭防火墙，关闭网络管理服务，关闭selinux

systemctl stop firewalld.service
systemctl disable firewalld.service
vi etc/selinux/config
setenforce 0

3.创建用户，用户组

groupadd -g 6001 oinstall
groupadd -g 6002 dba
groupadd -g 6003 oper
groupadd -g 6004 backupdba
groupadd -g 6005 dgdba
groupadd -g 6006 kmdba
groupadd -g 6007 asmdba
groupadd -g 6008 asmoper
groupadd -g 6009 asmadmin
groupadd -g 6010 racdba
useradd -u 6101 -g oinstall -G asmadmin,asmdba,dba,asmoper grid
useradd -u 6102 -g oinstall -G dba,backupdba,dgdba,kmdba,asmadmin,oper,asmoper,asmdba,racdba oracle
echo "oracle" | passwd --stdin grid
echo "oracle" | passwd --stdin oracle

4.配置本地yum源

echo "/dev/sr0 mnt/iso iso9660 defaults,loop 0 0" >> etc/fstab
mkdir -p mnt/iso
rm -rf etc/yum.repos.d/*
cat >> etc/yum.repos.d/ios.repo <<EOF
[ios]
name=ios
baseurl=file:///mnt/iso
enabled=1
gpgcheck=0
EOF
mount -a
yum repolist

5.创建目录

mkdir -p u01/app/{grid,oracle,19,soft,/oraInventory}
chown -R grid:oinstall    u01/app/{grid,19,oraInventory}
chown -R oracle:oinstall  u01/app/{oracle,soft}
chmod -R 775 u01/app

6.安装软件包

unset uninstall_rpm;for i in binutils compat-libstdc++-33  elfutils-libelf elfutils-libelf-devel elfutils-libelf-devel-static  gcc gcc-c++  glibc-devel glibc glibc-headers kernel-headers ksh libaio libaio-devel libgcc libgomp libstdc++ libstdc++-devel  make sysstat unixODBC unixODBC-devel  compat-gcc-34 compat-libstdc++-296 control-center glibc-common  libXp make openmotif22 setarch install compat-libcap1-1.10-1.x86_64 compat-libcap1-1.10-1.i686 smartmontools smartmontools net-tools cvuqdisk pdksh compat-libcap1 nfs-utils;do rpm -q $i &>/dev/null || uninstall_rpm="$uninstall_rpm $i";done ;[[ -z "$uninstall_rpm" ]] && echo -e "\nuninstall_rpm:\n\tOK.OK.OK" ||  echo -e "\nuninstall_rpm:\n\t$uninstall_rpm"
yum -y install lrzsz unzip openssh-clients scp rsync rlwrap

7.修改内核参数

/etc/sysctl.conf追加如下内容：

fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmax = 8184160256
kernel.shmall = 6547328205
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.panic_on_oops = 1

注：shmmax=total, shmall=total*0.8,shmmni > 4096

sysctl -p
sysctl -a

8.修改资源限制

/etc/security/limits.conf追加如下内容：

grid                soft    nproc  2047
grid                hard    nproc  16384
grid                soft    nofile 1024
grid                hard    nofile 65536
grid                soft  stack  10240
grid                hard  stack  32768
oracle              soft    nproc  2047
oracle              hard    nproc  16384
oracle              soft    nofile 1024
oracle              hard    nofile 65536
oracle              soft  stack  10240
oracle              hard  stack  32768
oracle              soft memlock 3145728
oracle              hard memlock 3145728

/etc/security/limits.d/20-nproc.conf追加如下内容：

* soft nproc 1024
* -    nproc 16384

/etc/pam.d/login追加如下内容：

session    required    pam_limits.so

9.修改主机名

[root@exn1 ~]# cat etc/hostname
exn1
[root@exn1 ~]# cat etc/sysconfig/network
# Created by anaconda
NETWORKING=yes
HOSTNAME=exn1
NOZEROCONF=yes
[root@exn1 ~]#

10.修改IO调度策略

[root@exn1 ~]# cat sys/block/sda/queue/scheduler
noop [deadline] cfq
[root@exn1 ~]# grubby --update-kernel=ALL --args="elevator=deadline"
[root@exn1 ~]# cat sys/block/sda/queue/scheduler
noop [deadline] cfq
[root@exn1 ~]# reboot        --重启生效

11.修改用户环境变量

grid:

exn1：

/home/grid/.bash_profile追加如下内容：

umask 022
export TMP=/tmp
export LANG=en_US
export TMPDIR=$TMP
export ORACLE_HOSTNAME=exn1
ORACLE_SID=+ASM1;export ORACLE_SID
ORACLE_TERM=xterm;export ORACLE_TERM
ORACLE_BASE=/u01/app/grid;export ORACLE_BASE
ORACLE_HOME=/u01/app/19;export ORACLE_HOME
NLS_DATE_FORMAT="yyyy-mm-ddHH24:MI:SS"; export NLS_DATE_FORMAT
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin;export PATH
THREADS_FLAG=native;export THREADS_FLAG

oracle:

exn1:

/home/oracle/.bash_profile追加如下内容：

export TMP=/tmp
export LANG=en_US
export TMPDIR=$TMP
export ORACLE_HOSTNAME=exn1
export ORACLE_UNQNAME=kevin
ORACLE_BASE=/u01/app/oracle;export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/19/db_1;export ORACLE_HOME
ORACLE_SID=kevin1;export ORACLE_SID
ORACLE_TERM=xterm;export ORACLE_TERM
NLS_DATE_FORMAT="yyyy-mm-ddHH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin;export PATH
THREADS_FLAG=native;export THREADS_FLAG

12.关闭透明大页

[root@exn1 ~]# cat sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@exn1 ~]# vi etc/default/grub
[root@exn1 ~]# cat etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
[root@exn1 ~]# grub2-mkconfig -o boot/grub2/grub.cfg   --重新生成配置文件
重启生效
[root@exn1 ~]# cat sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

停用系统时间同步服务，使用oracle CTSS时间服务

systemctl disable ntpd
systemctl stop ntpd
systemctl disable chronyd
systemctl stop chronyd
mv /etc/ndtp.conf /etc/ntpd.conf.bak
mv /etc/chrony.conf /etc/chrony.conf.bak

13.挂载共享存储

[root@exn1 ~]# yum -y install iscsi-initiator-utils  2>&1 1>/dev/null
[root@exn1 ~]# iscsiadm -m node -u all
[root@exn1 ~]# iscsiadm -m discovery -t st -p 192.168.2.137
[root@exn1 ~]# iscsiadm -m discovery -t st -p 192.168.2.138
[root@exn1 ~]# iscsiadm -m discovery -t st -p 192.168.2.139
[root@exn1 ~]# iscsiadm -m node -T 19C-SS0 -p 192.168.2.137 3260 -l
[root@exn1 ~]# iscsiadm -m node -T 19C-SS1 -p 192.168.2.138 3260 -l
[root@exn1 ~]# iscsiadm -m node -T 19C-SS2 -p 192.168.2.139 3260 -l
[root@exn1 Packages]# fdisk -l | grep /dev/sd
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   209715199   103808000   8e  Linux LVM
Disk /dev/sdb: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdc: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdd: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sde: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdf: 64.4 GB, 64424509440 bytes, 125829120 sectors
Disk /dev/sdg: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdh: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdi: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdj: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdk: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdl: 64.4 GB, 64424509440 bytes, 125829120 sectors
Disk /dev/sdm: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdn: 21.5 GB, 21474836480 bytes, 41943040 sectors
[root@exn1 Packages]# for i in b c d e f g h i j k l m n;
> do
> echo "KERNEL==\"sd?\", ENV{ID_SERIAL}==\"`/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", SYMLINK+=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\""  >> /etc/udev/rules.d/99-oracle-asmdevices.rules
> done
[root@exn1 Packages]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
[root@exn1 Packages]# cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45524f4d347056732d4250745a2d7747484a", SYMLINK+="ss0-vote", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45526a4642754b782d7a6b5a692d744e6956", SYMLINK+="ss1-vote", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552464c5a4537662d484e32752d666f7055", SYMLINK+="ss1-data1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45526b61526b31512d647338482d42464942", SYMLINK+="ss1-data2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552484a6c5175572d696564512d56676a47", SYMLINK+="ss1-gimr", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552505a654a32462d334d42332d394f6335", SYMLINK+="ss1-arch", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552796a347530442d315a44792d6a4e366d", SYMLINK+="ss1-flash", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552434c494570462d416e416b2d3674526c", SYMLINK+="ss2-vote", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45527a4b503079422d35476b792d514c4249", SYMLINK+="ss2-data1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45526263316f61662d706c5a6b2d41524a7a", SYMLINK+="ss2-data2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c45527973594b4a592d5a4f41412d64754f62", SYMLINK+="ss2-gimr", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c4552636a4b48355a2d4f69736a2d79314453", SYMLINK+="ss2-arch", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd?", ENV{ID_SERIAL}=="14f504e46494c455233416653367a2d6d4d47412d7646664a", SYMLINK+="ss2-flash", OWNER="grid", GROUP="asmadmin", MODE="0660"
[root@exn1 Packages]# scp /etc/udev/rules.d/99-oracle-asmdevices.rules exn2:/etc/udev/rules.d
[root@exn1 Packages]# scp /etc/udev/rules.d/99-oracle-asmdevices.rules exn3:/etc/udev/rules.d
[root@exn1 Packages]# scp /etc/udev/rules.d/99-oracle-asmdevices.rules exn4:/etc/udev/rules.d

[root@exn1 ~]# systemctl enable iscsi
[root@exn1 ~]# systemctl enable iscsid
Created symlink from /etc/systemd/system/multi-user.target.wants/iscsid.service to /usr/lib/systemd/system/iscsid.service.
[root@exn1 ~]# fdisk -l | grep /dev/sd
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   209715199   103808000   8e  Linux LVM
Disk /dev/sdb: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdc: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdd: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sde: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdf: 64.4 GB, 64424509440 bytes, 125829120 sectors
Disk /dev/sdg: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdh: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdi: 10.2 GB, 10234101760 bytes, 19988480 sectors
Disk /dev/sdj: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdk: 42.9 GB, 42949672960 bytes, 83886080 sectors
Disk /dev/sdl: 64.4 GB, 64424509440 bytes, 125829120 sectors
Disk /dev/sdm: 21.5 GB, 21474836480 bytes, 41943040 sectors
Disk /dev/sdn: 21.5 GB, 21474836480 bytes, 41943040 sectors
[root@exn1 ~]# partprobe /dev/sdb
[root@exn1 ~]# partprobe /dev/sdc
[root@exn1 ~]# partprobe /dev/sdd
[root@exn1 ~]# partprobe /dev/sde
[root@exn1 ~]# partprobe /dev/sdf
[root@exn1 ~]# partprobe /dev/sdg
[root@exn1 ~]# partprobe /dev/sdh
[root@exn1 ~]# partprobe /dev/sdi
[root@exn1 ~]# partprobe /dev/sdj
[root@exn1 ~]# partprobe /dev/sdk
[root@exn1 ~]# partprobe /dev/sdl
[root@exn1 ~]# partprobe /dev/sdm
[root@exn1 ~]# partprobe /dev/sdn
[root@exn1 ~]# ll /dev/ss*
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss0-vote -> sdb
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-arch -> sdg
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-data1 -> sdd
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-data2 -> sde
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-flash -> sdh
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-gimr -> sdf
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss1-vote -> sdc
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-arch -> sdm
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-data1 -> sdj
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-data2 -> sdk
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-flash -> sdn
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-gimr -> sdl
lrwxrwxrwx 1 root root 3 May 16 15:07 /dev/ss2-vote -> sdi
[root@exn1 ~]# ll /dev/sd*
brw-rw---- 1 root disk     8,   0 May 16 14:44 /dev/sda
brw-rw---- 1 root disk     8,   1 May 16 14:44 /dev/sda1
brw-rw---- 1 root disk     8,   2 May 16 14:44 /dev/sda2
brw-rw---- 1 grid asmadmin 8,  16 May 16 15:07 /dev/sdb
brw-rw---- 1 grid asmadmin 8,  32 May 16 15:07 /dev/sdc
brw-rw---- 1 grid asmadmin 8,  48 May 16 15:07 /dev/sdd
brw-rw---- 1 grid asmadmin 8,  64 May 16 15:07 /dev/sde
brw-rw---- 1 grid asmadmin 8,  80 May 16 15:07 /dev/sdf
brw-rw---- 1 grid asmadmin 8,  96 May 16 15:07 /dev/sdg
brw-rw---- 1 grid asmadmin 8, 112 May 16 15:07 /dev/sdh
brw-rw---- 1 grid asmadmin 8, 128 May 16 15:07 /dev/sdi
brw-rw---- 1 grid asmadmin 8, 144 May 16 15:07 /dev/sdj
brw-rw---- 1 grid asmadmin 8, 160 May 16 15:07 /dev/sdk
brw-rw---- 1 grid asmadmin 8, 176 May 16 15:07 /dev/sdl
brw-rw---- 1 grid asmadmin 8, 192 May 16 15:07 /dev/sdm
brw-rw---- 1 grid asmadmin 8, 208 May 16 15:07 /dev/sdn
[root@exn1 ~]#

14.上传压缩包检查完整性，并解压:

[root@exn1 ~]# cd /u01/app/soft/
[root@exn1 soft]# rz
[root@exn1 soft]# ll
total 5809468
-rw-r--r-- 1 root root 3059705302 May 16 15:19 LINUX.X64_193000_db_home.zip
-rw-r--r-- 1 root root 2889184573 Apr 28 20:35 LINUX.X64_193000_grid_home.zip
[root@exn1 soft]# sha256sum LINUX.X64_193000_grid_home.zip
d668002664d9399cf61eb03c0d1e3687121fc890b1ddd50b35dcbe13c5307d2e  LINUX.X64_193000_grid_home.zip
[root@exn1 soft]# sha256sum LINUX.X64_193000_db_home.zip
ba8329c757133da313ed3b6d7f86c5ac42cd9970a28bf2e6233f3235233aa8d8  LINUX.X64_193000_db_home.zip
[root@exn1 soft]#

如上，安装包完整.

[root@exn1 soft]# su - grid
Last login: Sat Apr 13 08:39:38 CST 2019 on pts/0
[grid@exn1 ~]$ cd /u01/app/soft/
[grid@exn1 soft]$ unzip linuxx64_12201_grid_home.zip -d $ORACLE_HOME
[grid@exn1 soft]$ exit
logout
[root@exn1 ~]# su - oracle
Last login: Sat Apr 13 08:39:27 CST 2019 on pts/0
[oracle@exn1 ~]$ cd /u01/app/soft/
[oracle@exn1 soft]$ mkdir /u01/app/oracle/product/19/db_1
[oracle@exn1 soft]$ unzip linuxx64_12201_database.zip  -d  $ORACLE_HOME

安装cvu软件包：

[root@exn1 R]# find / -name cvuqdisk-1.0.10-1.rpm
/tmp/CVU_19.0.0.0.0_grid/cvuqdisk-1.0.10-1.rpm
/tmp/GridSetupActions2019-05-16_04-15-18PM/CVU_19.0.0.0.0_grid/cvuqdisk-1.0.10-1.rpm
/u01/app/19/cv/remenv/cvuqdisk-1.0.10-1.rpm
/u01/app/19/cv/rpm/cvuqdisk-1.0.10-1.rpm
/u01/app/oracle/product/19/db_1/cv/remenv/cvuqdisk-1.0.10-1.rpm
/u01/app/oracle/product/19/db_1/cv/rpm/cvuqdisk-1.0.10-1.rpm
[root@exn1 R]# cd /u01/app/oracle/product/19/db_1/cv/rpm/
[root@exn1 rpm]# ll
total 12
-rw-r--r-- 1 oracle oinstall 11412 Mar 13 17:25 cvuqdisk-1.0.10-1.rpm
[root@exn1 rpm]# yum -y install cvuqdisk-1.0.10-1.rpm
w-r--r-- 1 oracle oinstall 11412 Mar 13 17:25 cvuqdisk-1.0.10-1.rpm
[root@exn1 rpm]# scp cvuqdisk-1.0.10-1.rpm  exn2:/u01/app/soft/
[root@exn1 rpm]# scp cvuqdisk-1.0.10-1.rpm  exn3:/u01/app/soft/
[root@exn1 rpm]# scp cvuqdisk-1.0.10-1.rpm  exn4:/u01/app/soft/

三.安装GI

最小化操作系统配置图形界面

yum install xorg-x11-apps
yum install xorg-x11-drivers

安装完成后，可用xshell直接调用图形。

配置site0,site1,site2三个中心。

site0作为仲裁中心，其它两个中心各放置两个集群节点。

配置磁盘组故障组时，分配两个故障组磁盘所属不同site机房。

按照提示，依次执行对应脚本。

节点1执行脚本，执行成功后，其他节点依次执行。

[root@exn1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@exn1 ~]# /u01/app/19/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/19
Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...




Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/exn1/crsconfig/rootcrs_exn1_2019-05-16_05-10-47PM.log
2019/05/16 17:10:55 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2019/05/16 17:10:55 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2019/05/16 17:10:55 CLSRSC-363: User ignored prerequisites during installation
2019/05/16 17:10:55 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2019/05/16 17:10:57 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2019/05/16 17:10:57 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2019/05/16 17:10:57 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2019/05/16 17:10:58 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2019/05/16 17:11:11 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2019/05/16 17:11:16 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2019/05/16 17:11:19 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2019/05/16 17:11:24 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2019/05/16 17:11:25 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2019/05/16 17:11:29 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2019/05/16 17:11:29 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2019/05/16 17:11:52 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2019/05/16 17:11:56 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2019/05/16 17:12:00 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2019/05/16 17:12:04 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
ASM has been created and started successfully.
[DBT-30001] Disk groups created successfully. Check /u01/app/grid/cfgtoollogs/asmca/asmca-190516PM051235.log for details.
2019/05/16 17:13:24 CLSRSC-482: Running command: '/u01/app/19/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk 22b081d103524fc5bffcde9f10b1fd6c.
Successful addition of voting disk 4964afe25e164f35bf48a9a6cbb60f11.
Successful addition of voting disk a328471af53f4faebf711f071d9bfdb0.
Successfully replaced voting disk group with +OCRDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
1. ONLINE   22b081d103524fc5bffcde9f10b1fd6c (/dev/ss1-vote) [OCRDG]
2. ONLINE   4964afe25e164f35bf48a9a6cbb60f11 (/dev/ss2-vote) [OCRDG]
3. ONLINE   a328471af53f4faebf711f071d9bfdb0 (/dev/ss0-vote) [OCRDG]
Located 3 voting disk(s).
2019/05/16 17:14:50 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2019/05/16 17:15:50 CLSRSC-343: Successfully started Oracle Clusterware stack
2019/05/16 17:15:50 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2019/05/16 17:17:36 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
[INFO] [DBT-30001] Disk groups created successfully. Check /u01/app/grid/cfgtoollogs/asmca/asmca-190516PM051742.log for details.




2019/05/16 17:18:35 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@exn1 ~]#

查看最后一步校验oracle环境报错信息。

完成GI安装

四. 配置asm磁盘组

创建datadg磁盘组

创建archdg磁盘组

创建flash磁盘组

确保所有磁盘组在所有节点正常挂载。

SQL> select group_number,name,value from v$asm_attribute where name like 'compatible%' order by group_number,name
GROUP_NUMBER NAME                 VALUE
------------ -------------------- ------------------------------
           1 compatible.asm       19.0.0.0.0
           1 compatible.rdbms     10.1.0.0.0
           2 compatible.asm       19.0.0.0.0
           2 compatible.rdbms     10.1.0.0.0
           3 compatible.asm       19.0.0.0.0
           3 compatible.rdbms     10.1.0.0.0
           4 compatible.asm       19.0.0.0.0
           4 compatible.rdbms     10.1.0.0.0
           5 compatible.asm       19.0.0.0.0
           5 compatible.rdbms     10.1.0.0.0
10 rows selected.
alter diskgroup archdg  set attribute 'compatible.rdbms'='19.0.0.0.0';
alter diskgroup datadg  set attribute 'compatible.rdbms'='19.0.0.0.0';
alter diskgroup flashdg set attribute 'compatible.rdbms'='19.0.0.0.0';
alter diskgroup ocrdg   set attribute 'compatible.rdbms'='19.0.0.0.0';
alter diskgroup gimrdg  set attribute 'compatible.rdbms'='12.2.0.0.0';         必须>=11.1.0.0
SQL> select group_number,name,value from v$asm_attribute where name like 'compatible%' order by group_number,name;
GROUP_NUMBER NAME                 VALUE
------------ -------------------- ------------------------------
           1 compatible.asm       19.0.0.0.0
           1 compatible.rdbms     19.0.0.0.0
           2 compatible.asm       19.0.0.0.0
           2 compatible.rdbms     19.0.0.0.0
           3 compatible.asm       19.0.0.0.0
           3 compatible.rdbms     19.0.0.0.0
           4 compatible.asm       19.0.0.0.0
           4 compatible.rdbms     12.2.0.0.0
           5 compatible.asm       19.0.0.0.0
           5 compatible.rdbms     19.0.0.0.0
10 rows selected.
SQL>

五. 安装数据库软件

[oracle@exn1 soft]$ cd $ORACLE_HOME
[oracle@exn1 db_1]$ ./runInstaller
ERROR: Unable to verify the graphical display setup. This application requires X display. Make sure that xdpyinfo exist under PATH variable.
Launching Oracle Database Setup Wizard...

执行脚本在所有节点。

[root@exn1 ~]# /u01/app/oracle/product/19/db_1/root.sh
Performing root user operation.
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/19/db_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@exn1 ~]#

六. 创建数据库

数据库安装完成。

七.校验

查看资源状态：

[grid@exn1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       exn1                     STABLE
               ONLINE  ONLINE       exn2                     STABLE
               ONLINE  ONLINE       exn3                     STABLE
               ONLINE  ONLINE       exn4                     STABLE
ora.chad
               ONLINE  ONLINE       exn1                     STABLE
               ONLINE  ONLINE       exn2                     STABLE
               ONLINE  ONLINE       exn3                     STABLE
               ONLINE  ONLINE       exn4                     STABLE
ora.net1.network
               ONLINE  ONLINE       exn1                     STABLE
               ONLINE  ONLINE       exn2                     STABLE
               ONLINE  ONLINE       exn3                     STABLE
               ONLINE  ONLINE       exn4                     STABLE
ora.ons
               ONLINE  ONLINE       exn1                     STABLE
               ONLINE  ONLINE       exn2                     STABLE
               ONLINE  ONLINE       exn3                     STABLE
               ONLINE  ONLINE       exn4                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ARCHDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.DATADG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.FLASHDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.GIMRDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       exn3                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       exn2                     169.254.18.48 192.16
                                                             8.2.192,STABLE
ora.OCRDG.dg(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     Started,STABLE
      2        ONLINE  ONLINE       exn2                     Started,STABLE
      3        ONLINE  ONLINE       exn3                     Started,STABLE
      4        ONLINE  ONLINE       exn4                     Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       exn1                     STABLE
      2        ONLINE  ONLINE       exn2                     STABLE
      3        ONLINE  ONLINE       exn3                     STABLE
      4        ONLINE  ONLINE       exn4                     STABLE
ora.cvu
      1        ONLINE  ONLINE       exn4                     STABLE
ora.exn1.vip
      1        ONLINE  ONLINE       exn1                     STABLE
ora.exn2.vip
      1        ONLINE  ONLINE       exn2                     STABLE
ora.exn3.vip
      1        ONLINE  ONLINE       exn3                     STABLE
ora.exn4.vip
      1        ONLINE  ONLINE       exn4                     STABLE
ora.kevin.db
      1        ONLINE  ONLINE       exn1                     Open,HOME=/u01/app/oracle/product/19/db_1,STABLE
      2        ONLINE  ONLINE       exn2                     Open,HOME=/u01/app/oracle/product/19/db_1,STABLE
      3        ONLINE  ONLINE       exn3                     Open,HOME=/u01/app/oracle/product/19/db1,STABLE
      4        ONLINE  ONLINE       exn4                     Open,HOME=/u01/app/oracle/product/19/db_1,STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       exn2                     Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       exn4                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       exn3                     STABLE
--------------------------------------------------------------------------------
[grid@exn1 ~]$

更多技术细节，欢迎关注公众号联系作者交流。

关注更多精彩

数据库

文章转载自MeetDB，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

19C Extended Rac Cluster 高可用架构搭建

评论