暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / Centos 7 系统 openGauss 3.1.0 一主两备集群在线扩容

Centos 7 系统 openGauss 3.1.0 一主两备集群在线扩容

原创 尚雷 2022-11-14
1187

注： 本次是在测试环境模拟一主两备集群库容新节点。

一、安装环境设置

1.1 硬件环境

名称 最低配置 建议配置 测试配置
服务器数量 3 略 略
硬盘 * 至少1GB用于安装openGauss的应用程序。
* 每个主机需大约300MB用于元数据存储。
* 预留70%以上的磁盘剩余空间用于数据存储。		 略 略
内存 >=2G 略 略
CPU 功能调试最小1×8核，2.0GHz 略 略
网络 300兆以上以太网，生产建议采用bond 略 略

1.2 软件环境

软件类型 信息描述
操作系统 Centos 7.6
Linux文件系统 剩余inode个数 > 15亿（推荐）
工具 bzip2
Python * oepnEuler：支持Python 3.7.x
* Centos 7.6: 支持Python 3.6.x
数据库版本 opengauss 3.1.0 企业版
数据库软件包名称 openGauss-3.1.0-CentOS-64bit-all.tar.gz
1.2.1 安装python
-- root用户【新增备库节点】
# 本次选择安装python 3.6.5版本
mkdir /usr/local/python3
cd /usr/local/python3
wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
tar -zxf Python-3.6.5.tgz
cd /usr/local/python3/Python-3.6.5
./configure --prefix=/usr/local/python3 --enable-shared CFLAGS=-fPIC  
make && make install
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

-- 设置环境变量
cat >>/etc/profile<<EOF
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/python3/bin
EOF
source /etc/profile
echo "/usr/local/python3/lib/" >> /etc/ld.so.conf
ldconfig

-- 如遇如下报错
python3: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
可执行 cp /usr/local/python3/Python-3.6.5/libpython3.6m.so.1.0 /usr/lib64/
1.2.2 安装软件依赖包
1.2.2.1 软件依赖要求
所需软件 建议版本
libaio-devel 建议版本：0.3.109-13
flex 要求版本：2.5.31 以上
bison 建议版本：2.7-4
ncurses-devel 建议版本：5.9-13.20130511
glibc-devel 建议版本：2.17-111
patch 建议版本：2.7.1-10
redhat-lsb-core 建议版本：4.1
readline-devel 建议版本 ：7.0-13
libnsl（openeuler+x86环境中） 建议版本 ：2.28-36
1.2.2.2 安装软件依赖包
-- root用户 【新增备库节点】
-- 安装依赖包
yum install -y libaio-devel flex bison ncurses-devel glibc-devel patch redhat-lsb-core readline-devel zlib readline gcc perl-ExtUtils-Embed readline-devel zlib-devel expect

-- 检查是否已安装
rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep -E "libaio-devel|flex|bison|ncurses-devel|glibc-devel|patch|redhat-lsb-core|readline-devel|zlib|readline|gcc|python|python-devel|perl-ExtUtils-Embed|readline-devel|zlib-devel|expect"
1.2.3 操作系统配置
1.2.3.1 修改操作系统参数
-- root用户 【新增备库节点】
-- 添加如下参数：
cat>>/etc/sysctl.conf <<EOF
net.ipv4.tcp_fin_timeout=60
net.ipv4.tcp_retries1=5
net.ipv4.tcp_syn_retries=5
net.sctp.path_max_retrans=10
net.sctp.max_init_retransmits=10
EOF

-- 生效参数
sysctl -p

## 执行sysctl -p 时有如下报错，此处选择忽略
sysctl: cannot stat /proc/sys/net/sctp/path_max_retrans: No such file or directory
sysctl: cannot stat /proc/sys/net/sctp/max_init_retransmits: No such file or directory

# 预安装过程根据系统参数配置会有相应提示，请按照提示对应系统参数
1.2.3.2 关闭透明大页
-- root用户 【新增备库节点】
-- openGauss默认关闭使用transparent_hugepage服务，并将关闭命令写入操作系统启动文件

cat >> /etc/rc.d/rc.local<<EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled;
	then
		echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag;
	then
		echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

-- 查看是否关闭：
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
1.2.3.3 关闭防火墙
-- root用户 【新增备库节点】
systemctl disable firewalld.service
systemctl stop firewalld.service
1.2.3.4 关闭selinux
-- root用户 【新增备库节点】
sed -i  "s/SELINUX=enforcing/SELINUX=disabled/"  /etc/selinux/config
-- 检查
cat /etc/selinux/config | grep disabled
1.2.3.5 修改字符集
-- root用户 【新增备库节点】
cat>> /etc/profile<<EOF
export LANG=en_US.UTF-8
EOF
source /etc/profile
# 检查
cat /etc/profile | grep LANG
env |grep -i lang
1.2.3.6 修改时区
-- root用户 【新增备库节点】
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
-- 查看
date
1.2.3.7 关闭SWAP
-- root用户 【新增备库节点】
-- 临时关闭
swapoff -a

-- 永久关闭
注释 swap 行
vim /etc/fstab

-- 查看
free -h 查看 swap一行全是 0 表示swap关闭
1.2.3.8 设置网卡MTU
-- root用户 【新增备库节点】
-- 设置网卡为1500
ifconfig 网卡名 mtu 1500

-- 查看
ifconfig -a |grep -i mtu
1.2.3.9 设置root用户远程登陆
-- root用户 【新增备库节点】
cat >>/etc/ssh/sshd_config<<EOF
PermitRootLogin yes
EOF
-- 检查
cat /etc/ssh/sshd_config |grep PermitRootLogin
1.2.3.10 修改 Banner 配置
-- root用户 【新增备库节点】
cat >>/etc/ssh/sshd_config<<EOF
Banner none
EOF
-- 检查
cat /etc/ssh/sshd_config | grep Banner

-- 重启生效
systemctl restart sshd.service
1.2.3.11 修改DNS配置
-- root用户 【新增备库节点】
-- 节点一
cat >>/etc/resolv.conf<<EOF
nameserver 192.168.17.112
EOF
1.2.3.12 添加hosts解析
-- root用户 【所有节点】
cat >> /etc/hosts<<EOF
192.168.17.112 opengauss-db4
EOF

二、集群规划

2.1 主机名-IP地址-端口号规划

项目名称 描述说明 备注
主机名 opengauss-db1、opengauss-db2、opengauss-db3、opengauss-db4 主备主机名
IP地址 192.168.17.113、192.168.17.139、192.168.17.159、192.168.17.112 主备主机名及IP
端口号 5000、26000 cm监听端口、DBnode监听端口

2.2 用户名规划

项目名称 名称 所属类型 规划建议
用户名 omm 操作系统 参照官网
组名 dbgrp 操作系统 参照官网

2.3 软件目录规划

目录名称 对应名称 目录作用
/opt/software/openGauss software 安装软件存放目录
/opt/gaussdb/install/app gaussdbAppPath 数据库安装目录
/opt/gaussdb/log gaussdbLogPath 日志目录
/opt/gaussdb/install/data/db1 dataNode1 主备节点数据存放目录
/opt/gaussdb/tmp tmpMppdbPath 临时文件目录
/opt/gaussdb/gausstools/om gaussdbToolPath 数据库工具目录
/opt/gaussdb/corefile corePath 数据库core文件目录
/opt/gaussdb/install/data/cm cmDir CM数据目录

三、扩容节点

3.1 扩容节点准备

3.1.1 创建用户
-- root用户 【新增备库节点】
-- 创建dbgrp用户组,组ID同主库
[root@opengauss-db4 ~]# /usr/sbin/groupadd -g 1004 dbgrp
-- 创建omm用户，用户ID同主库
[root@opengauss-db4 ~]# /usr/sbin/useradd -u 1003 -g dbgrp -G dbgrp omm
-- 设置omm用户密码
[root@opengauss-db4 ~]# echo "omm123" | passwd --stdin omm
3.1.2 配置互信
-- 配置互信，配置root用户及omm用户互信

-- 配置 root 用户互信【主节点 root用户执行】
[root@opengauss-db1 ~]# cd /opt/software/openGauss/script/
[root@opengauss-db1 script]# vim hostfile   -- 在该目录下编辑所有节点IP文件，参照如下
192.168.17.113
192.168.17.139
192.168.17.159
192.168.17.112
-- 执行gs_sshexkey 配置互信
[root@opengauss-db1 script]# ./gs_sshexkey -f hostfile
Please enter password for current user[root].  
Password:      -- 输入 root口令
Checking network information.
All nodes in the network are Normal.
Successfully checked network information.
Creating SSH trust.
Creating the local key file.
Successfully created the local key files.
Appending local ID to authorized_keys.
Successfully appended local ID to authorized_keys.
Updating the known_hosts file.
Successfully updated the known_hosts file.
Appending authorized_key on the remote node.
Successfully appended authorized_key on all remote node.
Checking common authentication file content.
Successfully checked common authentication content.
Distributing SSH trust file to all node.
Distributing trust keys file to all node successfully.
Successfully distributed SSH trust file to all node.
Verifying SSH trust on all hosts.
Successfully verified SSH trust on all hosts.
Successfully created SSH trust.

-- 测试root用户互信
[root@opengauss-db1 script]# ssh 192.168.17.112
[root@opengauss-db1 script]# ssh 192.168.17.139
[root@opengauss-db1 script]# ssh 192.168.17.159

-- 配置 omm 用户互信 【主节点 omm 用户执行】
[omm@opengauss-db1 ~]$ cd /u01/app/software/script/
[omm@opengauss-db1 script]$ ./gs_sshexkey -f hostfile
Please enter password for current user[omm].
Password:
Checking network information.
All nodes in the network are Normal.
Successfully checked network information.
Creating SSH trust.
Creating the local key file.
Successfully created the local key files.
Appending local ID to authorized_keys.
Successfully appended local ID to authorized_keys.
Updating the known_hosts file.
Successfully updated the known_hosts file.
Appending authorized_key on the remote node.
Successfully appended authorized_key on all remote node.
Checking common authentication file content.
Successfully checked common authentication content.
Distributing SSH trust file to all node.
Distributing trust keys file to all node successfully.
Successfully distributed SSH trust file to all node.
Verifying SSH trust on all hosts.
Successfully verified SSH trust on all hosts.
Successfully created SSH trust.

-- 测试omm用户互信
[omm@opengauss-db1 script]$ ssh 192.168.17.112
[omm@opengauss-db1 script]$ ssh 192.168.17.139
[omm@opengauss-db1 script]$ ssh 192.168.17.159

-- gs_sshexkey 详细语法可查看 gs_sshexkey -? | --help
3.1.3 配置新增节点环境变量
-- 拷贝已有节点omm用户环境变量文件至新增节点
[omm@opengauss-db1 ~]$ scp .bashrc omm@192.168.17.112:/home/omm/
    
[omm@opengauss-db1 ~]$ scp .bash_profile omm@192.168.17.112:/home/omm/

-- 修改新增节点 omm用户.bashrc, 删除 或 注释 export GAUSS_ENV=2
# GAUSS_ENV 值为2表示安装成功 

-- 生效新增节点 omm 用户环境变量
[omm@opengauss-db4 ~]$ source .bashrc
[omm@opengauss-db4 ~]$ source .bash_profil

-- 拷贝已有节点root用户环境变量文件至新增节点
[root@opengauss-db1 ~]# scp -r /etc/profile root@192.168.17.112:/etc/profile

-- 生效新增节点root用户环境变量
[root@opengauss-db4 ~]# source /etc/profile
3.1.4 修改XML配置文件
3.1.4.1 配置XML文件
-- 主节点 root 用户
-- 在 /opt/software/openGauss 目录下修改 clusterconfig.xml配置文件
-- 执行操作如下
cat > clusterconfig.xml<<EOF
<?xml version="1.0" encoding="UTF-8"?>
<ROOT>
    <!-- openGauss整体信息 -->
    <CLUSTER>
    <!-- 数据库名称 -->
        <PARAM name="clusterName" value="openGSDB" />
    <!-- 数据库节点名称(hostname) -->
    <!-- 添加新增节点主机名 -->
        <PARAM name="nodeNames" value="opengauss-db1,opengauss-db2,opengauss-db3,opengauss-db4" />
    <!-- 节点IP，与nodeNames一一对应 -->
    <!-- 添加新增节点IP -->
        <PARAM name="backIp1s" value="192.168.17.113,192.168.17.139,192.168.17.159,192.168.17.112"/>
    <!-- 数据库安装目录-->
        <PARAM name="gaussdbAppPath" value="/opt/gaussdb/install/app" />
    <!-- 日志目录-->
        <PARAM name="gaussdbLogPath" value="/opt/gaussdb/log" />
    <!-- 临时文件目录-->
        <PARAM name="tmpMppdbPath" value="/opt/gaussdb/tmp"/>
    <!--数据库工具目录-->
        <PARAM name="gaussdbToolPath" value="/opt/gaussdb/gausstools/om" />
    <!--数据库core文件目录-->
        <PARAM name="corePath" value="/opt/gaussdb/corefile"/>
    <!-- openGauss类型，此处示例为单机类型，"single-inst"表示单机一主多备部署形态-->
        <PARAM name="clusterType" value="single-inst"/>
    </CLUSTER>
    <!-- 每台服务器上的节点部署信息 -->
    <DEVICELIST>
        <!-- opengauss-db1上的节点部署信息 -->
        <DEVICE sn="1000001">
        <!-- opengauss-db1的hostname -->
            <PARAM name="name" value="opengauss-db1"/>
        <!-- opengauss-db1所在的AZ及AZ优先级 -->
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
        <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.113"/>
            <PARAM name="sshIp1" value="192.168.17.113"/>
            
        <!--CM-->
     <!--CM数据目录-->
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
            <PARAM name="cmsNum" value="1" />
      <!--CM监听端口-->
            <PARAM name="cmServerPortBase" value="5000" />
            <PARAM name="cmServerlevel" value="1" />
      <!-- CM所有实例所在节点名及监听ip -->
      <!-- 添加新增节点IP -->
            <PARAM name="cmServerListenIp1" value="192.168.17.113,192.168.17.139,192.168.17.159,192.168.17.112" />
      <!-- 添加新增节点主机名 -->      
            <PARAM name="cmServerRelation" value="opengauss-db1,opengauss-db2,opengauss-db3,opengauss-db4" />
            
      <!-- dbnode -->
        <PARAM name="dataNum" value="1"/>
      <!--DBnode端口号-->
        <PARAM name="dataPortBase" value="26000"/>
      <!-- DBnode主节点上数据目录，及备机数据目录 -->
      <!-- 添加新增节点主机名及数据目录 -->
        <PARAM name="dataNode1" value="/opt/gaussdb/install/data/db1,opengauss-db2,/opt/gaussdb/install/data/db1,opengauss-db3,/opt/gaussdb/install/data/db1,opengauss-db4,/opt/gaussdb/install/data/db1"/>
      <!-- DBnode节点上设定同步模式的节点数 -->
        <PARAM name="dataNode1_syncNum" value="0"/>
        </DEVICE>

        <!-- opengauss-db2上的节点部署信息，其中"name"的值配置为主机名称(hostname) -->
        <DEVICE sn="1000002">
            <PARAM name="name" value="opengauss-db2"/>
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
            <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.139"/>
            <PARAM name="sshIp1" value="192.168.17.139"/>
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
        </DEVICE>

        <!-- opengauss-db3上的节点部署信息，其中"name"的值配置为主机名称(hostname) -->
        <DEVICE sn="1000003">
            <PARAM name="name" value="opengauss-db3"/>
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
            <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.159"/>
            <PARAM name="sshIp1" value="192.168.17.159"/>
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
        </DEVICE>
        
        <!-- 添加新增节点部署信息 -->
		<!-- opengauss-db4上的节点部署信息，其中"name"的值配置为主机名称(hostname) -->
		<!-- 设置新增节点sn等信息 -->
        <DEVICE sn="1000004">
            <PARAM name="name" value="opengauss-db4"/>
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
            <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.112"/>
            <PARAM name="sshIp1" value="192.168.17.112"/>
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
        </DEVICE>
    </DEVICELIST>
</ROOT>
EOF
3.1.4.2 参数简介
实例类型 参数 说明
整体信息 name 主机名称
azName 指定azName（Available Zone Name），字符串（不能含有特殊字符），例如AZ1、AZ2、AZ3。
azPriority 指定azPriority的优先级。
backIp1 主机在后端存储网络中的IP地址（内网IP）。所有openGauss主机使用后端存储网络通讯。
sshIp1 设置SSH可信通道IP地址（外网IP）。若无外网，则可以不设置该选项或者同backIp1设置相同IP。
3.1.5 拷贝主节点clusterconfig.xml至新增节点
-- 新增节点创建 /opt/software/openGauss 目录
[root@opengauss-db4 ]# mkdir -p /opt/software/openGauss
-- 拷贝主节点 clusterconfig.xml 文件至新增节点同目录处
[root@opengauss-db1 ]# scp /opt/software/openGauss/clusterconfig.xml omm@opengauss-db4:/opt/software/openGauss

3.2 集群扩容

-- 主节点 【root 用户执行】
[omm@opengauss-db1 ~]$ su - root
Password: 
Last login: Fri Nov 11 15:14:54 CST 2022 from 192.168.16.10 on pts/0
[root@opengauss-db1 ~]# cd /opt/software/openGauss/script
[root@opengauss-db1 script]# source /home/omm/.bashrc
[root@opengauss-db1 script]# ./gs_expansion -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -h 192.168.17.112

执行结果如下：
Start expansion with cluster manager component.
Start to send soft to each standby nodes.
End to send soft to each standby nodes.
Success to send XML to new nodes
Start to perform perinstall on nodes: ['opengauss-db4']
Preinstall command is: /tmp/gs_expansion_2022-11-13_21_31_49_537605/pkg/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -L  --non-interactive 2>&1
Success to perform perinstall on nodes ['opengauss-db4']
Success to change user to [omm]
Installing applications on all new nodes.
Install on new node output: [SUCCESS] opengauss-db4:
Using omm:dbgrp to install database.
Using installation program path : /opt/gaussdb/install/app
Command for creating symbolic link: ln -snf /opt/gaussdb/install/app_4e931f9a /opt/gaussdb/install/app.
Decompressing bin file.
Decompress CM package command: export LD_LIBRARY_PATH=$GPHOME/script/gspylib/clib:$LD_LIBRARY_PATH && tar -zxf "/opt/gaussdb/gausstools/om/script/os_platform/./../../openGauss-3.1.0-CentOS-64bit-cm.tar.gz" -C "/opt/gaussdb/install/app"
Decompress CM package successfully.
Successfully decompressed bin file.
Modifying Alarm configuration.
Modifying user's environmental variable $GAUSS_ENV.
Successfully modified user's environmental variable $GAUSS_ENV.
Fixing file permission.
Set Cgroup config file to appPath.
Successfully Set Cgroup.

Successfully installed APP on nodes ['opengauss-db4'].
success to send all CA file.
Success to change user to [omm]
Success to init instance on nodes ['opengauss-db4']
Start to generate and send cluster static file.
End to generate and send cluster static file.

Ready to perform command on node [opengauss-db4]. Command is : source /home/omm/.bashrc;gs_guc set -D /opt/gaussdb/install/data/db1 -h 'host    all    omm    192.168.17.112/32    trust' -h 'host    all    all    192.168.17.112/32    sha256'
Successfully set hba on all nodes.
Success to change user to [omm]
Stopping cluster.
=========================================
Successfully stopped cluster.
=========================================
End stop cluster.
Remove dynamic_config_file and CM metadata directory on all nodes.
Starting cluster.
======================================================================
Successfully started primary instance. Wait for standby instance.
======================================================================
.
Successfully started cluster.
======================================================================
cluster_state      : Normal
redistributing     : No
node_count         : 4
Datanode State
    primary           : 1
    standby           : 3
    secondary         : 0
    cascade_standby   : 0
    building          : 0
    abnormal          : 0
    down              : 0

Expansion results:
192.168.17.112: Success

-- 执行过程日志可查看 /opt/gaussdb/log/omm/pg_log/dn__xxx 目录下 postgresql 日志

3.3 查询集群状态

[root@opengauss-db1 script]# su - omm
Last login: Sun Nov 13 21:37:57 CST 2022 on pts/4
[omm@opengauss-db1 ~]$ cm_ctl query -v -C -i -d
[  CMServer State   ]

node             node_ip         instance                                    state
------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  1    /opt/gaussdb/install/data/cm/cm_server Primary
2  opengauss-db2 192.168.17.139  2    /opt/gaussdb/install/data/cm/cm_server Standby
3  opengauss-db3 192.168.17.159  3    /opt/gaussdb/install/data/cm/cm_server Standby
4  opengauss-db4 192.168.17.112  4    /opt/gaussdb/install/data/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node             node_ip         instance                           state            | node             node_ip         instance                           state            | node             node_ip         instance                           state            | node             node_ip         instance                           state
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  6001 /opt/gaussdb/install/data/db1 P Primary Normal | 2  opengauss-db2 192.168.17.139  6002 /opt/gaussdb/install/data/db1 S Standby Normal | 3  opengauss-db3 192.168.17.159  6003 /opt/gaussdb/install/data/db1 S Standby Normal | 4  opengauss-db4 192.168.17.112  6004 /opt/gaussdb/install/data/db1 S Standby Normal


[omm@opengauss-db1 ~]$ gs_om -t status --detail
[  CMServer State   ]

node             node_ip         instance                                    state
------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  1    /opt/gaussdb/install/data/cm/cm_server Primary
2  opengauss-db2 192.168.17.139  2    /opt/gaussdb/install/data/cm/cm_server Standby
3  opengauss-db3 192.168.17.159  3    /opt/gaussdb/install/data/cm/cm_server Standby
4  opengauss-db4 192.168.17.112  4    /opt/gaussdb/install/data/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node             node_ip         instance                           state            
-------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  6001 /opt/gaussdb/install/data/db1 P Primary Normal
2  opengauss-db2 192.168.17.139  6002 /opt/gaussdb/install/data/db1 S Standby Normal
3  opengauss-db3 192.168.17.159  6003 /opt/gaussdb/install/data/db1 S Standby Normal
4  opengauss-db4 192.168.17.112  6004 /opt/gaussdb/install/data/db1 S Standby Normal

3.4 刷新动态配置文件

-- 主节点 omm 用户操作
[omm@opengauss-db1 ~]$ gs_om -t refreshconf
Generating dynamic configuration file for all nodes.
Successfully generated dynamic configuration file.

3.5 数据测试

-- 主节点
[root@opengauss-db1 ~]# su - omm
Last login: Mon Nov 14 10:14:24 CST 2022 on pts/5
[omm@opengauss-db1 ~]$ gsql -d postgres -p 26000
gsql ((openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:24 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=# create table pgtb (id int, name text);
CREATE TABLE
openGauss=# insert into pgtb (id,name) values (10,'Jacky');
INSERT 0 1
openGauss=# 

-- 新增节点
[omm@opengauss-db4 ~]$ gsql -d postgres -p 26000
gsql ((openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:24 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=# select * from pgtb;
 id | name  
----+-------
 10 | Jacky
(1 row)

openGauss=# insert into pgtb (id,name) values (11,'shlei6067');
ERROR:  cannot execute INSERT in a read-only transaction
openGauss=#

四、补充

4.1 扩容时报GAUSS-51100

--问题现象
主节点执行扩容时，报 [GAUSS-51100] : Failed to verify SSH trust on these nodes错误
[root@opengauss-db1 openGauss]# ./script/gs_expansion -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -h 192.168.17.112
[GAUSS-51100] : Failed to verify SSH trust on these nodes: 
opengauss-db1, opengauss-db2, opengauss-db3, opengauss-db4, 192.168.17.113, 192.168.17.139, 192.168.17.159, 192.168.17.112 by root
opengauss-db1, opengauss-db2, opengauss-db3, opengauss-db4, 192.168.17.113, 192.168.17.139, 192.168.17.159, 192.168.17.112 by individual user.

-- 解决办法
创建存放所有节点IP文件，分别在root用户及omm用户下参照3.1.2章节执行 gs_sshexkey 来配置节点间的互信，并分别测试互信
可在主节点执行 /opt/gaussdb/gausstools/om/script/gspylib/pssh/bin/pssh -s -H <hostname> 'pwd' 查看互信

# gs_expansion文件倒数第二行expansion.checkTrust()是用来做互信校验的

4.2 解析clusterconfig.xml报错

-- 问题现象：
主节点执行扩容时报如下错误：
[root@opengauss-db1 script]# ./gs_expansion -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -h 192.168.17.112
Start expansion with cluster manager component.
Start to send soft to each standby nodes.
End to send soft to each standby nodes.
Success to send XML to new nodes
Start to perform perinstall on nodes: ['opengauss-db4']
Preinstall command is: /tmp/gs_expansion_2022-11-13_21_21_21_185901/pkg/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -L  --non-interactive 2>&1
Success to perform perinstall on nodes ['opengauss-db4']
Success to change user to [omm]
Installing applications on all new nodes.
Install on new node output: [FAILURE] opengauss-db4:
Using omm:dbgrp to install database.
Using installation program path : /opt/gaussdb/install/app
Command for creating symbolic link: ln -snf /opt/gaussdb/install/app_4e931f9a /opt/gaussdb/install/app.
[GAUSS-51234] : The configuration file [/opt/software/openGauss/clusterconfig.xml] contains parsing errors. Error:
[GAUSS-51236] : Failed to parsing xml. Error: 
[Errno 13] Permission denied: '/opt/software/openGauss/clusterconfig.xml'.

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/python3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/software/openGauss/script/impl/expansion/expansion_impl_with_cm.py", line 446, in do_install
    self.install_app()
  File "/opt/software/openGauss/script/impl/expansion/expansion_impl_with_cm.py", line 172, in install_app
    [key for key in result_map.keys() if result_map[key] == "Failure"])
Exception: [GAUSS-52707] : Failed to install on ['opengauss-db4'].

-- 解决办法：
可将主节点clusterconfig.xml文件拷贝至新增备库相同目录节点下，并赋予omm用户读权限，参照 3.1.5 章节
该问题主要是由于 3.1.0 版本在扩容时拷贝文件到新增节点，clusterconfig.xml 用户和属组都是 root，omm用户无法读该文件

4.3 扩容时报GAUSS-51802

-- 问题现象：
[GAUSS-51802] : Failed to obtain the environment variable "GAUSSHOME", please import environment variable.

-- 解决办法：
主节点root用户执行 source /home/omm/.bashrc
新增节点 omm 用户执行 source /home/omm/.bashrc

4.4 扩容时报GAUSS-51800

-- 问题现象：
扩容时，报如下错误：
[root@opengauss-db1 script]# source /home/omm/.bashrc
[root@opengauss-db1 script]# ./script/gs_expansion -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -h 192.168.17.112
-bash: ./script/gs_expansion: No such file or directory
[root@opengauss-db1 script]# ./gs_expansion -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -h 192.168.17.112
Start expansion with cluster manager component.
Start to send soft to each standby nodes.
End to send soft to each standby nodes.
Success to send XML to new nodes
Start to perform perinstall on nodes: ['opengauss-db4']
Preinstall command is: /tmp/gs_expansion_2022-11-13_20_49_42_523008/pkg/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml -L  --non-interactive 2>&1
Success to perform perinstall on nodes ['opengauss-db4']
Success to change user to [omm]
Installing applications on all new nodes.
Install on new node output: [FAILURE] opengauss-db4:
Using omm:dbgrp to install database.
Using installation program path : /opt/gaussdb/install/app
[GAUSS-51800] : The environmental variable $GAUSSHOME is empty. or variable has exceeded maximum length

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/python3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/software/openGauss/script/impl/expansion/expansion_impl_with_cm.py", line 446, in do_install
    self.install_app()
  File "/opt/software/openGauss/script/impl/expansion/expansion_impl_with_cm.py", line 172, in install_app
    [key for key in result_map.keys() if result_map[key] == "Failure"])
Exception: [GAUSS-52707] : Failed to install on ['opengauss-db4'].

-- 解决办法：
新增节点 omm 用户 修改 /home/omm/.bashrc ,取消或注释 GAUSS_ENV 
然后执行 source /home/omm/.bashrc 生效环境变量

4.5 主节点/opt/gaussdb/gausstools/ 目录为空

-- 问题现象：
在扩容时，发现主节点 /opt/gaussdb/gausstools/ 目录为空

-- 解决办法：
使用 omm 用户 拷贝其它正常备库节点 /opt/gaussdb/gausstools/ 目录至 主节点 omm用户 /opt/gaussdb/gausstools/ 下
墨力计划opengauss训练营opengauss
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论