第二章：如何部署 OceanBase 社区版 2.7 如何使用 OBD 自动化部署多节点集群

部署规划

本文介绍 OceanBase 三节点集群的部署方法，该部署方法需要通过中控机直接远程登录到 OceanBase 节点上部署启动 observer 和 obproxy 进程。

机器信息

机器和角色划分

磁盘划分

此处使用 LVM 技术对 /dev/vdb 进行划分，需要登录到每个节点上手动初始化。

# lvm 分盘
pvcreate /dev/vdb
vgcreate obvg /dev/vdb
lvcreate obvg -L 20G^C
lvcreate -L 20G obvg -n lvredo
lvcreate -l 100%FREE obvg -n lvdata

# 格式化文件系统
mkfs.ext4 /dev/obvg/lvdata
mkfs.ext4 /dev/obvg/lvredo

# 修改 mount 参数文件
vim /etc/fstab
/dev/obvg/lvredo          /redo              ext4            defaults,noatime,nodiratime,nodelalloc,barrier=0        0 0
/dev/obvg/lvdata             /data         ext4            defaults,noatime,nodiratime,nodelalloc,barrier=0        0 0

# 挂载文件系统
mkdir -p /data /redo
vim /etc/fstab
mount /data
mount /redo
chown -R admin:admin /data /redo

# 检查
df -h

输出：

文件系统                 容量  已用  可用 已用% 挂载点
/dev/mapper/obvg-lvdata   59G   53M   56G    1% /data
/dev/mapper/obvg-lvredo   20G   45M   19G    1% /redo

编辑 OBD 配置文件

OBD 针对不同的部署场景提供不同的配置文件。这些配置文件示例放在 OceanBase 开源项目地址中，您可访问链接查看：https://github.com/oceanbase/obdeploy/tree/master/example。

如果您是部署三节点版本，只需下载如下两个配置文件：

• 部署三节点 observer 进程：

  https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-example.yaml

• 部署三节点 observer 和 obproxy 进程：

  https://github.com/oceanbase/obdeploy/blob/master/example/mini-distributed-with-obproxy-example.yaml

  说明

  本文以 3.1.0 版本的 OceanBase 和 ODP 为例介绍如何部署，若您使用 V3.2.3 及以上版本的 OBProxy，需将下述配置文件中的 obproxy 组件名改为 obproxy-ce。

此处仿照生产环境，选择第二种部署配置文件下载。

[admin@obce00 ~]$ cat obce-3zones.yaml

#  Only need to configure when remote login is required

user:
   username: admin
#    password: your password if need
   key_file: /home/admin/.ssh/id_rsa.pub
   port: your ssh port, default 22
#    timeout: ssh connection timeout (second), default 30
oceanbase-ce:
  servers:
    - name: obce01
      # Please don't use hostname, only IP can be supported
      ip: 172.20.249.52
    - name: obce02
      ip: 172.20.249.49
    - name: obce03
      ip: 172.20.249.51
  global:
    # Please set devname as the network adaptor's name whose ip is  in the setting of severs.
    # if set severs as "127.0.0.1", please set devname as "lo"
    # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
    devname: eth0
    cluster_id: 2
    # please set memory limit to a suitable value which is matching resource.
    memory_limit: 8G # The maximum running memory for an observer
    system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    stack_size: 512K
    cpu_count: 16
    cache_wash_threshold: 1G
    __min_full_resource_pool_memory: 268435456
    workers_per_cpu_quota: 10
    schema_history_expire_time: 1d
    # The value of net_thread_count had better be same as cpu's core number.
    net_thread_count: 4
    major_freeze_duty_time: Disable
    minor_freeze_times: 10
    enable_separate_sys_clog: 0
    enable_merge_by_turn: FALSE
    #datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
    datafile_size: 50G
    syslog_level: WARN # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 10 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
    # observer cluster name, consistent with obproxy's cluster_name
    appname: obce-3zones
    root_password: 0E****8d # root user password, can be empty
    proxyro_password: uY****zx # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
  obce01:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo
    zone: zone1
  obce02:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo
    zone: zone2
  obce03:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo
    zone: zone3
obproxy:
  servers:
    - 172.20.249.52
    - 172.20.249.49
    - 172.20.249.51
  # Set dependent components for the component.
  # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
  depends:
    - oceanbase-ce
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/obproxy
    # oceanbase root server list
    # format: ip:mysql_port;ip:mysql_port
    rs_list: 172.20.249.52:2881;172.20.249.49:2881;172.20.249.51:2881
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname
    cluster_name: obce-3zones
    obproxy_sys_password: 0M****tm # obproxy sys user password, can be empty
    observer_sys_password: uY****zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty

该配置文件是专门针对最小内存（可用内存大于 8G）的节点配置，里面指定了很多 observer 进程的启动参数。您需注意 yaml 的格式，每个配置项后面冒号（:) 跟后面的值之间必须有个空格（' ')。 下面对关键的几个参数进行补充说明下：

部署成功后，OBD 会将配置文件 obce-3zones.yaml 复制到自己的工作目录里（~/.obd/cluster/obce-3zones/config.yaml），后期再对 obce-3zones.yaml 文件进行修改是不生效的。

OBD 部署集群

配置文件准备好后，就可以部署该配置文件对应的集群，部署内容主要包含：

• 复制软件到相应节点，并安装软件。

• 在相应节点创建相关目录。

您可使用命令 obd cluster deploy [集群名] -c 集群配置文件 进行部署。

[admin@obce00 ~]$ obd cluster deploy obce-3zones -c obce-3zones.yaml
oceanbase-ce-3.1.0 already installed.
obproxy-3.1.0 already installed.
+-----------------------------------------------------------------------------+
|                                   Packages                                  |
+--------------+---------+---------+------------------------------------------+
| Repository   | Version | Release | Md5                                      |
+--------------+---------+---------+------------------------------------------+
| oceanbase-ce | 3.1.0   | 3.el8   | 84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 |
| obproxy      | 3.1.0   | 1.el8   | d242ea5fe45222b8f61c3135ba2aaa778c61ea22 |
+--------------+---------+---------+------------------------------------------+
Repository integrity check ok
Parameter check ok
Open ssh connection ok
Remote oceanbase-ce-3.1.0-84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 repository install ok
Remote oceanbase-ce-3.1.0-84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80 repository lib check ok
Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository install ok
Remote obproxy-3.1.0-d242ea5fe45222b8f61c3135ba2aaa778c61ea22 repository lib check ok
Cluster status check ok
Initializes cluster work home ok
Initializes cluster work home ok
obce-3zones deployed

检查集群部署状态。

[admin@obce00 ~]$ obd cluster list
+----------------------------------------------------------------------+
|                             Cluster List                             |
+-------------+--------------------------------------+-----------------+
| Name        | Configuration Path                   | Status (Cached) |
+-------------+--------------------------------------+-----------------+
| obce-3zones | /home/admin/.obd/cluster/obce-3zones | deployed        |
+-------------+--------------------------------------+-----------------+

OBD 启动和初始化集群

上面 deploy 操作只是安装了软件和准备初始化目录，您还需使用命令 obd cluster start 启动集群节点并初始化集群。

obd cluster start obce-3zones

输出：
[admin@obce00 ~]$ obd cluster start obce-3zones
Get local repositories and plugins ok
Open ssh connection ok
Cluster param config check ok
Check before start observer ok
[WARN] (172.20.249.52) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[WARN] (172.20.249.52) The recommended number of open files is 655350 (Current value: 65535)
[WARN] (172.20.249.49) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[WARN] (172.20.249.49) The recommended number of open files is 655350 (Current value: 65535)
[WARN] (172.20.249.51) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[WARN] (172.20.249.51) The recommended number of open files is 655350 (Current value: 65535)

Check before start obproxy ok
Start observer ok
observer program health check ok
Connect to observer ok
Initialize cluster
Cluster bootstrap ok
Wait for observer init ok
+-------------------------------------------------+
|                     observer                    |
+---------------+---------+------+-------+--------+
| ip            | version | port | zone  | status |
+---------------+---------+------+-------+--------+
| 172.20.249.49 | 3.1.0   | 2881 | zone2 | active |
| 172.20.249.51 | 3.1.0   | 2881 | zone3 | active |
| 172.20.249.52 | 3.1.0   | 2881 | zone1 | active |
+---------------+---------+------+-------+--------+

Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize cluster
+-------------------------------------------------+
|                     obproxy                     |
+---------------+------+-----------------+--------+
| ip            | port | prometheus_port | status |
+---------------+------+-----------------+--------+
| 172.20.249.52 | 2883 | 2884            | active |
| 172.20.249.49 | 2883 | 2884            | active |
| 172.20.249.51 | 2883 | 2884            | active |
+---------------+------+-----------------+--------+
obce-3zones running

如果集群节点内核参数和会话限制参数不符合要求，安装时会给出提示。

这个命令会进行几分钟的 bootstrap。当可用内存不足 8G 或者日志目录剩余可用空间比例不足 5% 的时候，这个 bootstrap 是很可能会失败的。

确认集群是否初始化成功（可选）

这个步骤是可选的。第一次学习或生产部署时，建议检查一下。

• 查看启动后的集群状态

  [admin@obce00 ~]$ obd cluster list
  +----------------------------------------------------------------------+
  |                             Cluster List                             |
  +-------------+--------------------------------------+-----------------+
  | Name        | Configuration Path                   | Status (Cached) |
  +-------------+--------------------------------------+-----------------+
  | obce-3zones | /home/admin/.obd/cluster/obce-3zones | running         |
  +-------------+--------------------------------------+-----------------+

• 检查 OceanBase 集群各个节点进程信息

  OceanBase 数据库是单进程软件，进程名为 observer ，可运行以下命令查看这个进程。

  IPS="172.20.249.52 172.20.249.49 172.20.249.51"
  for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done

  输出：

  [admin@obce00 oceanbase-ce]$ for ob in $IPS;do echo $ob; ssh $ob "ps -ef | grep observer | grep -v grep "; done
  172.20.249.52
  admin       6987       1 69 08:35 ?        01:38:26 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0E****8d,redo_dir=/redo -z zone1 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN
  172.20.249.49
  admin       7064       1 87 08:35 ?        02:02:59 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0E****8d,redo_dir=/redo -z zone2 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN
  172.20.249.51
  admin       6920       1 72 08:35 ?        01:42:42 /home/admin/oceanbase-ce/bin/observer -r 172.20.249.52:2882:2881;172.20.249.49:2882:2881;172.20.249.51:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=3G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,major_freeze_duty_time=Disable,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=10,root_password=0E****8d,redo_dir=/redo -z zone3 -p 2881 -P 2882 -n obce-3zones -c 2 -d /data -i eth0 -l WARN

  从进程里看，可执行文件是 /home/admin/oceanbase-ce/bin/observer ，实际上它是个软链接。

  [admin@obce00 oceanbase-ce]$ ll /home/admin/oceanbase-ce/bin/observer
  lrwxrwxrwx 1 admin admin 100 Sep 11 17:16 /home/admin/oceanbase-ce/bin/observer -> /home/admin/.obd/repository/oceanbase-ce/3.1.0/84bd2fe27f8b8243cc57d8a3f68b4c50f94aab80/bin/observer

  进程启动的时候，通过 -o 指定了很多参数，这些参数都是在前面 OBD 集群部署配置文件里指定的。

• 检查 OceanBase 集群各个节点监听状况

  IPS="172.20.249.52 172.20.249.49 172.20.249.51"
  for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done
  
  输出：
  [admin@obce00 ~]$ for ob in $IPS;do echo $ob; ssh $ob "netstat -ntlp"; done
  172.20.249.52
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
  tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      6987/observer
  tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      6987/observer
  tcp        0      0 0.0.0.0:2883            0.0.0.0:*               LISTEN      7640/obproxy
  tcp        0      0 0.0.0.0:2884            0.0.0.0:*               LISTEN      7640/obproxy
  172.20.249.49
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      7064/observer
  tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      7064/observer
  tcp        0      0 0.0.0.0:2883            0.0.0.0:*               LISTEN      7718/obproxy
  tcp        0      0 0.0.0.0:2884            0.0.0.0:*               LISTEN      7718/obproxy
  tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
  172.20.249.51
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  Active Internet connections (only servers)
  Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
  tcp        0      0 0.0.0.0:2881            0.0.0.0:*               LISTEN      6920/observer
  tcp        0      0 0.0.0.0:2882            0.0.0.0:*               LISTEN      6920/observer
  tcp        0      0 0.0.0.0:2883            0.0.0.0:*               LISTEN      7574/obproxy
  tcp        0      0 0.0.0.0:2884            0.0.0.0:*               LISTEN      7574/obproxy
  tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
  
  
  <p data-line="407" class="sync-line" style="margin:0;"></p>

连接 OceanBase 集群的内部实例（sys)

传统的 MySQL 客户端可以连接 OceanBase 社区版，前提是 MySQL 的版本是 5.5/5.6/5.7。OceanBase 数据库也提供自己的客户端工具 OBClient，需要安装后才可使用。 和传统 MySQL 不同的是 ODP 的连接端口是 2883， 连接用户名为 root@sys#集群名，密码则在前面 OBD 配置文件里指定。

mysql -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0E****8d -c -A oceanbase

输出：
[admin@obce00 ~]$ mysql -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0E****8d -c -A oceanbase
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.6.25 OceanBase 3.1.0 (r3-b20901e8c84d3ea774beeaca963c67d7802e4b4e) (Built Aug 10 2021 08:10:38)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> show databases;
+--------------------+
| Database           |
+--------------------+
| oceanbase          |
| information_schema |
| mysql              |
| SYS                |
| LBACSYS            |
| ORAAUDITOR         |
| test               |
+--------------------+
7 rows in set (0.002 sec)

MySQL [oceanbase]> select a.zone,concat(a.svr_ip,':',a.svr_port) observer, cpu_total, (cpu_total-cpu_assigned) cpu_free, round(mem_total/1024/1024/1024) mem_total_gb, round((mem_total-mem_assigned)/1024/1024/1024) mem_free_gb, usec_to_time(b.last_offline_time) last_offline_time, usec_to_time(b.start_service_time) start_service_time, b.status, usec_to_time(b.stop_time) stop_time, b.build_version
from __all_virtual_server_stat a join __all_server b on (a.svr_ip=b.svr_ip and a.svr_port=b.svr_port)
order by a.zone, a.svr_ip
;

+-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+
| zone  | observer           | cpu_total | cpu_free | mem_total_gb | mem_free_gb | last_offline_time          | start_service_time         | status | stop_time                  | build_version                                                          |
+-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+
| zone1 | 172.20.249.52:2882 |        14 |     11.5 |            5 |           4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:06.357140 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) |
| zone2 | 172.20.249.49:2882 |        14 |     11.5 |            5 |           4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:07.605244 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) |
| zone3 | 172.20.249.51:2882 |        14 |     11.5 |            5 |           4 | 1970-01-01 08:00:00.000000 | 2021-09-12 08:36:07.631981 | active | 1970-01-01 08:00:00.000000 | 3.1.0_3-b20901e8c84d3ea774beeaca963c67d7802e4b4e(Aug 10 2021 08:10:38) |
+-------+--------------------+-----------+----------+--------------+-------------+----------------------------+----------------------------+--------+----------------------------+------------------------------------------------------------------------+
3 rows in set (0.004 sec)

当在数据库列表里看到 oceanbase 数据库时就表示集群初始化成功。

OBClient 安装和使用示例

sudo rpm -ivh /tmp/obd/obclient-*.rpm /tmp/obd/libobclient-*.rpm

obclient -h 172.20.249.52 -uroot@sys#obce-3zones -P2883 -p0E****8d -c -A oceanbase