YashanDB共享集群初体验

概述

YashanDB共享集群基于YashanDB内核持续演进，是一个单库多实例的多活数据库系统，支持在线故障自动切换和故障自动恢复，核心组件主要包括

• 崖山集群内核（YCK，Yashan Cluster Kernel）
• 崖山集群服务（YCS，Yashan Cluster Service）
• 崖山文件系统（YFS，Yashan File System）

1、服务器规划

2、服务器配置

2.1 参数调整

资源限制：

vi /etc/security/limits.conf

* soft nofile 1048576
* hard nofile 1048576
* soft nproc 1048576
* hard nproc 1048576
* soft rss unlimited
* hard rss unlimited
* soft stack 8192
* hard stack 8192

配置大页内存：

测试环境，内存较小，略

关闭透明大页：

测试环境，略

关闭防火墙：

systemctl stop firewalld
systemctl disable firewalld

2.2 创建安装用户

创建安装用户：

groupadd YASDBA
useradd -d /home/yashan -m -G YASDBA yashan
passwd yashan

配置sudo免密：

vi /etc/sudoers

yashan  ALL=(ALL)NOPASSWD:ALL

2.3 创建安装目录

mkdir -p /data/yashan
chmod -R 777 /data/yashan

2.4 绑定共享磁盘

由于vbox虚拟机无法获取UUID，需要改虚拟机的配置文件，所以这里采用另外一种方式，通过ATTR的方式绑定

# fdisk对共享盘进行分区
fdisk /dev/sdb
fdisk /dev/sdc
fdisk /dev/sdd
...
...

Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    10485759     5241856   83  Linux

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048    10485759     5241856   83  Linux

Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048    41943039    20970496   83  Linux

# 通过udevadm获取磁盘size信息
[root@yashanr1 dev]# udevadm info -a -p /sys/block/sdb/sdb1 | grep -e "ATTR{start}" -e "ATTR{size}"
    ATTR{size}=="10483712"
    ATTR{start}=="2048"
[root@yashanr1 dev]# udevadm info -a -p /sys/block/sdc/sdc1 | grep -e "ATTR{start}" -e "ATTR{size}"
    ATTR{size}=="10483712"
    ATTR{start}=="2048"
[root@yashanr1 dev]# udevadm info -a -p /sys/block/sdd/sdd1 | grep -e "ATTR{start}" -e "ATTR{size}"
    ATTR{size}=="41940992"
    ATTR{start}=="2048"

# 配置rule文件
[root@yashanr1 dev]# cat /etc/udev/rules.d/yashan-device-rule.rules
KERNEL=="sdb1",SUBSYSTEM=="block", ATTR{size}=="10483712", ATTR{start}=="2048",OWNER="yashan",SYMLINK+="yas/ycr",GROUP="YASDBA", MODE="0666"
KERNEL=="sdc1",SUBSYSTEM=="block", ATTR{size}=="10483712", ATTR{start}=="2048",OWNER="yashan",SYMLINK+="yas/vote",GROUP="YASDBA", MODE="0666"
KERNEL=="sdd1",SUBSYSTEM=="block", ATTR{size}=="41940992", ATTR{start}=="2048",OWNER="yashan",SYMLINK+="yas/data",GROUP="YASDBA", MODE="0666"

# 重启udev
systemctl restart systemd-udev-trigger.service

# 检查确认，两节点均正常识别
[root@yashanr1 dev]# ll /dev/sd* | grep yashan
brw-rw-rw-. 1 yashan YASDBA 8, 17 Nov 26 12:15 /dev/sdb1
brw-rw-rw-. 1 yashan YASDBA 8, 33 Nov 26 12:15 /dev/sdc1
brw-rw-rw-. 1 yashan YASDBA 8, 49 Nov 26 12:15 /dev/sdd1
[root@yashanr1 dev]# ll /dev/yas/*
lrwxrwxrwx. 1 root root 7 Nov 26 12:15 /dev/yas/data -> ../sdd1
lrwxrwxrwx. 1 root root 7 Nov 26 12:15 /dev/yas/vote -> ../sdc1
lrwxrwxrwx. 1 root root 7 Nov 26 12:15 /dev/yas/ycr -> ../sdb1

3、集群部署

3.1 解压安装包

[yashan@yashanr1 ~]$ cd install/
[yashan@yashanr1 install]$ ls
yashandb-23.2.4.100-linux-x86_64.tar.gz
[yashan@yashanr1 install]$ tar -zxf yashandb-23.2.4.100-linux-x86_64.tar.gz
[yashan@yashanr1 install]$ ll
total 185408
drwxrwxr-x. 6 yashan yashan        70 Sep  4 17:29 admin
drwxrwxr-x. 2 yashan yashan       243 Sep  4 17:29 bin
drwxrwxr-x. 2 yashan yashan       176 Sep  4 17:28 conf
drwxrwxr-x. 4 yashan yashan        33 Sep  4 17:28 ext
-rw-rw-r--. 1 yashan yashan     10701 Sep  4 17:28 gitmoduleversion.dat
drwxrwxr-x. 2 yashan yashan        64 Sep  4 17:28 include
drwxrwxr-x. 3 yashan yashan        17 Sep  4 17:29 java
drwxrwxr-x. 2 yashan yashan      4096 Sep  4 17:29 lib
-rw-r-----. 1 yashan yashan     14989 Sep  4 17:28 LICENSE
drwxrwxr-x. 3 yashan yashan        21 Sep  4 17:29 plug-in
drwxrwxr-x. 2 yashan yashan        61 Sep  4 17:29 scripts
-rw-r--r--. 1 root   root   189823431 Nov 26 13:39 yashandb-23.2.4.100-linux-x86_64.tar.gz

3.2 生成配置文件

注意，需要内存4g以上，openssl 1.1.1及以上，否则部署失败。

升级openssl：

tar -zxvf  openssl-1.1.1k.tar.gz
cd openssl-1.1.1k
./config --prefix=/usr/local/openssl
make && make install
mv /usr/bin/openssl /usr/bin/openssl.bak
ln -s /usr/local/openssl/bin/openssl /usr/bin/openssl
ln -s /usr/local/openssl/include/openssl /usr/include/openssl
echo "/usr/local/openssl/lib/" >> /etc/ld.so.conf
ldconfig -v
openssl version

生成配置文件：

[yashan@yashanr1 install]$ ./bin/yasboot package ce gen --cluster yashanrac -u yashan -p xjlove1224 --ip 192.168.56.30,192.168.56.31 --port 22 --install-path /data/yashan/yasdb_home --data-path /data/yashan/yasdb_data --begin-port 1688 --node 2 --data /dev/yas/data --vote /dev/yas/vote  --ycr /dev/yas/ycr

 hostid   | group | node_type | node_name | listen_addr        | inter_connect      | data_path
--------------------------------------------------------------------------------------------------------------
 host0001 | ceg1  | ce        | 1-1       | 192.168.56.30:1688 | 192.168.56.30:1689 | /data/yashan/yasdb_data
----------+-------+-----------+-----------+--------------------+--------------------+-------------------------
 host0002 | ceg1  | ce        | 1-2       | 192.168.56.31:1688 | 192.168.56.31:1689 | /data/yashan/yasdb_data
----------+-------+-----------+-----------+--------------------+--------------------+-------------------------

Generate config completed


# 生成两个配置文件，根据实际需要调整yashandb.toml配置文件中的安装参数
[yashan@yashanr1 install]$ ll | grep toml
-rw-------. 1 yashan yashan       835 Nov 26 13:44 hosts.toml
-rw-------. 1 yashan yashan      2277 Nov 26 13:44 yashanrac.toml

# 这里简单调整了几个参数：
REDO_FILE_NUM = 6
REDO_FILE_SIZE = "512M"

SHM_POOL_SIZE = "1G"
SYS_AREA_SIZE = "512M"

3.3 部署集群

[yashan@yashanr1 install]$ ./bin/yasboot package install -t hosts.toml -i yashandb-23.2.4.100-linux-x86_64.tar.gz

checking install package...
install version: yashandb 23.2.4.100
host0001 100% [====================================================================]    3s
host0002 100% [====================================================================]    3s
update host to yasom...
...
...

3.4 部署数据库

[yashan@yashanr1 install]$ ./bin/yasboot cluster deploy -t yashanrac.toml --yfs-force-create
 type | uuid             | name               | hostid | index     | status  | return_code | progress | cost
-------------------------------------------------------------------------------------------------------------
 task | c5ac179ecb16cda2 | DeployYasdbCluster | -      | yashanrac | SUCCESS | 0           | 100      | 114
------+------------------+--------------------+--------+-----------+---------+-------------+----------+------
task completed, status: SUCCESS

3.5 配置环境变量

[yashan@yashanr1 ~]$ cd /data/yashan/yasdb_home/yashandb/23.2.4.100/conf/
[yashan@yashanr1 conf]$ ls
database_options.json  monit  profile.toml  sqlcollect.toml  sqlhtml.template  yashanrac.bashrc  yasreport.template  ycs_options.json  yfs_options.json
[yashan@yashanr1 conf]$ cat yashanrac.bashrc >>  ~/.bashrc
[yashan@yashanr1 conf]$ source ~/.bashrc
[yashan@yashanr1 conf]$

3.6 设置SYS密码

[yashan@yashanr1 install]$ yasboot cluster password set -n xjlove_1224 -c yashanrac
 type | uuid             | name             | hostid | index     | status  | return_code | progress | cost
-----------------------------------------------------------------------------------------------------------
 task | 66e234138cce2672 | YasdbPasswordSet | -      | yashanrac | SUCCESS | 0           | 100      | 2
------+------------------+------------------+--------+-----------+---------+-------------+----------+------
task completed, status: SUCCESS

3.7 集群检查

[yashan@yashanr1 install]$ yasboot cluster status -c yashanrac -d
 hostid   | node_type | nodeid | pid   | instance_status | database_status | database_role | listen_address     | data_path
-------------------------------------------------------------------------------------------------------------------------------------------------
 host0001 | ce        | 1-1:1  | 10179 | open            | normal          | primary       | 192.168.56.30:1688 | /data/yashan/yasdb_data/ce-1-1
----------+-----------+--------+-------+-----------------+-----------------+---------------+--------------------+--------------------------------
 host0002 | ce        | 1-2:2  | 13652 | open            | normal          | primary       | 192.168.56.31:1688 | /data/yashan/yasdb_data/ce-1-2
----------+-----------+--------+-------+-----------------+-----------------+---------------+--------------------+--------------------------------

SQL> select status,host_name,instance_name from gv$instance;

STATUS        HOST_NAME                                                        INSTANCE_NAME
------------- ---------------------------------------------------------------- ----------------------------------------------------------------
OPEN          yashanr1                                                         yasdb
OPEN          yashanr2                                                         yasdb

2 rows fetched.


# 使用ycsctl需要配置YASCS_HOME环境变量
[yashan@yashanr1 yasdb_data]$ ycsctl status
---------------------------------------------------------------------------------------------
Self Host ID|Cluster Master ID|YasFS Master ID|YasDB Master ID|Active Host Count
---------------------------------------------------------------------------------------------
1            1                 1               1               2
---------------------------------------------------------------------------------------------
Host ID   |Target    |State     |YasFS     |YasDB
---------------------------------------------------------------------------------------------
1          online     online     online     online
2          online     online     online     online

4、集群基础管理

4.1 集群状态检查

[yashan@yashanr1 ~]$ ycsctl status
---------------------------------------------------------------------------------------------
Self Host ID|Cluster Master ID|YasFS Master ID|YasDB Master ID|Active Host Count
---------------------------------------------------------------------------------------------
1            1                 1               1               2
---------------------------------------------------------------------------------------------
Host ID   |Target    |State     |YasFS     |YasDB
---------------------------------------------------------------------------------------------
1          online     online     online     online
2          online     online     online     online

# State字段表示YCS服务的当前状态。
# YasFS字段表示YFS服务的当前状态。
# YasDB字段表示数据库实例的当前状态。
# Target字段表示数据库实例是否纳入YCS一起启动，根据$YASCS_HOME/config/yascs.ini中参数AUTO_START而定，当AUTO_START=ALWAYS时，Target为online，当AUTO_START=NEVER时，Target为offline

4.2 集群启停

# 停止集群
[yashan@yashanr1 ~]$ yasboot cluster stop -c yashanrac
 type | uuid             | name             | hostid | index     | status  | return_code | progress | cost
-----------------------------------------------------------------------------------------------------------
 task | 071b464226c4a71b | StopYasdbCluster | -      | yashanrac | SUCCESS | 0           | 100      | 16
------+------------------+------------------+--------+-----------+---------+-------------+----------+------
task completed, status: SUCCESS

# 启动集群
[yashan@yashanr1 ~]$ yasboot cluster start -c yashanrac
 type | uuid             | name              | hostid | index     | status  | return_code | progress | cost
------------------------------------------------------------------------------------------------------------
 task | 83dcb4bfce3f0d16 | StartYasdbCluster | -      | yashanrac | SUCCESS | 0           | 100      | 18
------+------------------+-------------------+--------+-----------+---------+-------------+----------+------
task completed, status: SUCCESS

# 重启集群
[yashan@yashanr1 ~]$ yasboot cluster restart -c yashanrac
 type | uuid             | name                | hostid | index     | status  | return_code | progress | cost
--------------------------------------------------------------------------------------------------------------
 task | 62af87902ff9c147 | ReStartYasdbCluster | -      | yashanrac | SUCCESS | 0           | 100      | 36
------+------------------+---------------------+--------+-----------+---------+-------------+----------+------
task completed, status: SUCCESS